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Executive Summary 


American Institutes for Research (AIR) conducted an evaluation of the effect on struggling 
readers of implementing the READ 180 reading intervention in five participating schools in 
Milwaukee Public Schools (MPS) under a Striving Readers grant from the U.S. Department of 
Education. The evaluation used an experimental design in order to produce a rigorous estimate of 
the impact of the READ 180 intervention on measures of reading achievement for struggling 
students. The evaluation also explored implementation fidelity and the contexts and conditions of 
implementation that may extend or limit the intervention’s effects. 


Methodology 


To measure program impact on students’ academic performance in reading, AIR analyzed 
student achievement data collected from the Northwest Evaluation Association (NWEA) 
Measures of Academic Progress (MAP) benchmark assessment. AIR also administered a student 
survey to assess the impact on student engagement and self-efficacy for reading. 


To measure program fidelity, AIR developed rubrics for assigning implementation fidelity scores 
for both the professional development model and the READ 180 classroom model. Each rubric 
was based on data collected from multiple sources, including interviews with READ 180 
teachers and building administrators, classroom observations, and extant data student and 
teacher-level data. Scores were categorized into low, medium, and high levels of implementation 
for each READ 180 teacher/classroom.' Descriptive analyses of implementation data also were 
conducted to provide context for implementation successes and challenges. 


Findings 


Research Question One: Does the READ 180 reading intervention improve students’ 
academic performance in reading? 


The program had a significant impact on students’ academic performance in reading. Students 
assigned to READ 180 scored approximately 1.8 points higher than control students when 
controlling for pretest scores and student-level covariates. This represents an effect size of d= 
0.138. 


Overall, results of the student survey were inconclusive, with students from the control and 
treatment groups responding similarly to the majority of survey items, though treatment students 
did indicate slightly higher self-efficacy in reading as well as higher levels of behavioral 
engagement. Due to the small sample size of study students participating in the survey, no 
differences were found to be statistically significant. 


' The unit of analysis for the implementation fidelity analysis was to be the teacher. Because of teacher tummover 
during the year, however, this report will refer to teachers as classrooms. 
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Research Question Two: With what fidelity did the program implement the professional 
development model and what factors mediated the level of implementation? 


Overall, the professional development model was implemented with medium fidelity. Some 
components of the model were implemented and attended as planned, including the READ 180 
orientation trainings. However, other components were not. For example, some teachers did not 
complete the online RED Course trainings and three of the nine READ 180 roundtable sessions 
were cancelled. In addition, teachers did not receive individual mentoring sessions as planned. 


Research Question Three: With what fidelity did classroom intervention teachers 
implement READ 180 and what factors mediated the level of implementation? 


Overall, the READ 180 classroom model was implemented with high fidelity. However, student 
attendance in READ 180 classrooms was low and students’ use of the READ 180 instructional 
software ranged from medium to low levels of implementation. Therefore, nearly all classrooms 
received a medium rating for fidelity to the classroom model. Teachers reported that prior 
experience teaching READ 180 and the support of the district READ 180 coordinator were 
important facilitators for successfully implementing the model. 
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Introduction and Study Background 


American Institutes for Research (AIR) conducted an evaluation of the effect on struggling 
readers of implementing the READ 180 reading intervention in five participating schools in 
Milwaukee Public Schools (MPS) under a Striving Readers grant from the U.S. Department of 
Education. The purpose of this evaluation was to use an experimental design in order to produce 
a rigorous estimate of the impact of the READ 180 intervention on measures of reading 
achievement for struggling students. The evaluation also explored implementation fidelity and 
the contexts and conditions of implementation that may extend or limit the intervention’s effects. 
The evaluation assessed the implementation and impact of the READ 180 program on struggling 
readers in the five participating schools over the one year of grant implementation, using a 
mixed-method approach and multiple data sources~. The primary purpose was to produce a 
rigorous causal impact estimate. The secondary purpose was to assess the fidelity of program 
implementation in the study schools. 


READ 180 Overview 


READ 180 is a comprehensive reading intervention program developed by Scholastic. The 
program is designed to address individual student needs through adaptive instructional software, 
high-interest literature, and direct instruction in reading and writing skills. The READ 180 
program follows a structured instructional model. Each class period begins with 20 minutes of 
whole-group instruction. Next, students break into three small groups and spend 20 minutes in 
each of the following rotations: small-group instruction, instructional software, and modeled and 
independent reading. The READ 180 class ends with time to reflect on what was learned that day 
during a 10-minute whole-group wrap-up. READ 180 was implemented in the five participating 
schools as part of Striving Readers in Fall 2010 through Spring 2011. 


General Overview of the Striving Readers Evaluation 


The evaluation focused on three primary research questions and used data collected from a 
variety of sources to measure implementation and impact. The data sources included student 
assessments, student surveys, extant professional development attendance records, classroom 
observations, READ 180 software logs from the Scholastic Achievement Manager (SAM), and 
interviews with intervention teachers and building principals. Table 1 presents the alignment 
between the evaluation measures and the research questions. A detailed description of the plans 
for evaluating the implementation and impact of the intervention, are provided in subsequent 
sections of this report. 


* The original evaluation was designed to assess the impact of READ 180 over two years of intervention; however, 
funding for subsequent years of the program and the evaluation was eliminated. 
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Table 1. Alignment of Research Questions to Evaluation Measures 


Research Questions 


Measures 


1. Does the READ 180 reading intervention improve students’ 
academic performance in reading? 

a. What is the impact of the READ 180 intervention on struggling 
Grade 6-10 students’ academic performance in reading? 

b. Are there differences between the amount of impact of the 
READ 180 intervention on the academic performance in 
reading of middle school students (Grades 6—8) and the amount 
of impact on that of high school students (Grades 9-10)? 

c. Are there differences between the amount of impact of the 
READ 180 intervention on the academic performance in 
reading of students with identified special needs and the 
amount of impact on that of other struggling readers? 

d. Does the READ 180 reading intervention increase student 
engagement and self-efficacy related to reading? 


e NWEA MAP assessment 
e WKCE 
e Student survey 


2. With what fidelity did the program implement the professional 
development model and what factors mediated the level of 
implementation? 

a. To what extent were professional development opportunities 
provided and attended as planned? 

b. How effective were professional development opportunities in 
preparing teachers to implement READ 180 and appropriate 
literacy instructional content and strategies? (In other words, 
what was the perceived quality of the professional 
development? What was the perceived impact on teachers’ 
behaviors? Do teachers feel prepared and comfortable 
implementing the READ 180 classroom model?) 

c. To what extent were the identified curricular materials and 
resources made available to intervention teachers as planned? 


e Professional development 
logs 

e Teacher interviews 

e Principal interviews 


3. With what fidelity did classroom intervention teachers implement 
READ 180 and what factors mediated the level of 
implementation? 

a. To what extent did intervention teachers implement the READ 
180 classroom model with fidelity? 

b. Did the students assigned to the treatment group attend the 
READ 180 class as planned? 


American Institutes for Research 


Classroom observations 
Teacher interviews 
Principal interviews 
Scholastic Achievement 
Manager (SAM) Time-on- 
Task log 

e Period attendance 
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Part I: Intervention and Logic Model 


This section provides graphic and narrative description of the intervention and its expected 
connections with student outcomes. The program was designed to provide the participating 
schools with a series of resources, including instructional materials, newly hired literacy 
intervention teachers, and professional development opportunities. If these resources were 
provided as planned, the logic model predicts that the intervention teachers would become 
knowledgeable about the READ 180 classroom model as a supplemental literacy class for 
struggling readers and become comfortable implementing it. In addition, READ 180 resources 
and supports were expected to enhance intervention teachers’ knowledge and use of effective 
instructional practices. Ultimately, the use of effective classroom practices should lead to 
changes in students’ behavior and academic performance. The following subsections provide 
detail about the specific components of the intervention model, including descriptions of the 
students targeted by the intervention and the characteristics of READ 180 teachers and 
classrooms. 


Description of the Intervention Model 


READ 180 is a comprehensive reading intervention program developed by Scholastic. The 
program is designed to address individual student needs through adaptive instructional software, 
high-interest literature, and direct instruction in reading and writing skills. The READ 180 
program follows a structured instructional model. Each class period begins with 20 minutes of 
whole-group instruction. Next, students break into three small groups and spend 20 minutes in 
each of the following rotations: small-group instruction, instructional software, and modeled and 
independent reading. The READ 180 class ends with time to reflect on what was learned that day 
during a 10-minute whole-group wrap-up. 


The primary text used during whole-group and small-group instruction is the student rBook, 
which is an interactive work text. The rBook is organized into nine topical instructional 
workshops. Each workshop contains six sections: Preview/Teach Vocabulary, Reading, 
Vocabulary/Word Study, Writing and Grammar, Functional Literacy, and a Workshop Wrap-Up. 
Throughout the workshop, there are checkpoints that allow teachers to assess student knowledge 
and provide opportunities for more in-depth skill instruction and practice using the Resources for 
Differentiated Instruction text. READ 180 also provides teachers with a variety of assessment 
resources to inform instruction: Scholastic Achievement Manager (SAM) Time-on-Task Report, 
rSkills Test Reports, Scholastic Reading Inventory (SRI) Reports, and Reading Counts! Reports. 


Students Targeted by the Intervention 
During the 2010-11 school year, READ 180 was implemented schoolwide in Grades 6-9 in five 


Title I schools: Audubon School, James Madison Academic Campus (J-MAC), Mitchell School, 
Morse School for Gifted and Talented, and Northwest Secondary School. 
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Eligibility Criteria 


The guidelines set forth by Milwaukee Public Schools for entrance to the READ 180 program 
determined a student’s eligibility for random assignment according to whether he or she met one 
or more of the following criteria: 


e The student scores Minimal or Basic on the Wisconsin Knowledge and Concepts 
Examination (WKCE) during fall 2009. 


e WKCE data are not available for a student, and that student scores Minimal or Basic on 
the Discovery Education Assessment Predictive Benchmark Assessment (i.e., ThinkLink) 
on the most recently administered assessment (February—March 2010 or May 2010). 


e No assessment data are available for the student, and teacher assessments and 
observations indicate that the student is performing at least two grade levels below 
expectations—that is, a Student Promotion System (SPS) rating of 1 on the 1-4 rating 
scale. 


Students with disabilities also were eligible for the intervention if they have completed one year 
of the Language! course (meaning they will be at level C or above during the upcoming school 
year). In addition, English language learner (ELL) students who have a Language Acquisition 
Unit (LAU) level of 3.0 or higher were eligible for the intervention. ° 


Exclusionary Criteria 


Students were excluded from the intervention if they meet any of the following criteria: 


e Students with disabilities who are classified as Language level A during fall 2009 and 
students who have not completed the Language! course for an entire year will not be 
included in the study. In addition, students with disabilities participating in Unique (who 
take the state’s alternative assessment) will be excluded from the sample. 


e ELL students at an LAU level below 3.0 will be excluded. 


e Students who are receiving English as a second language (ESL) services may be 
excluded from the sample because they cannot be pulled from their ESL services to 
participate in the supplemental reading intervention. Those students who are able to 
enroll without conflict in both ESL services and the supplemental reading intervention 
can be included in the study. 


e Students who are eligible for the sample on one or more student assessments but are 
deemed proficient/advanced by a principal or teacher will be excluded. 


3 In September 2010, MPS informed Learning Point Associates (merged with AIR in January 2011) that the criterion 
level for English language proficiency was raised to limit the eligible students to only those students with a LAU 
level of 4.0 and above. 
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Selecting READ 180 Teachers 


As Table 2 shows, eight reading intervention teachers were hired to teach three to six 90-minute 
reading blocks daily serving approximately 450—500 students in the treatment group. Seven of 
the eight teachers were assigned to a single school and one teacher split time between James 
Madison Academic and Northwest Secondary School. These teachers were distinct from the 
regular ELA teachers providing instruction in the five study schools. No interaction was 
anticipated between the eight reading intervention teachers and the control students. Table 2 
shows the planned distribution of sections across the schools as cohorts of students progressed 
through the three years. The Striving Readers grant was implemented only during the 2010-11 


school year. 


Table 2. Distribution of READ 180 Teachers and Students By School 


Number of Sections 


School Grade 2010-11 | 2011-12 | 2012-13 | Number of Teachers 
6 2 2 
7 2 1 
Audubon 8 2 1 2 
9 1 
10 1 
1 2 
2 1 1 
Morse 1 2 1 2 
2 1 1 
10 2 1 
JMAC Z z : 2/3 
10 2 
1 2 
2 1 2 
Northwest Secondary 2 2 1 21/3 
2 2 1 
10 2 1 
1 1 1 
Mitchell 1 1 1 1 
1 1 1 
Total 24 24 24 8 
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MPS advertised for and hired the READ 180 intervention teachers through the school district 
before the beginning of the 2010—11 school year. The project coordinator trained and helped 
them transition into their assigned school. MPS developed a job posting that outlined the 
following qualifications and educational requirements that successful candidates must possess: 


¢ A minimum of three (3) years successful teaching experience within MPS or successful 
teaching experience in READ 180 while fully certified with a valid Wisconsin teaching 
license 


¢ Licensed Reading Specialist/Teacher (316 or 317) or able to complete its licensing 
requirements within two (2) years of being hired as a READ 180 teacher 


* Knowledge of current instructional practices related to a standards-based instructional 
design, including alignment to the Wisconsin Model Academic Standards 


* Strong reading background, with experience in READ 180 preferred 
¢ Experience working with at-risk middle or high school students 
¢ Experience in diverse, multicultural teaching and learning environments 


* Demonstrated ability to work as a collaborative and flexible team member in addressing 
school issues and challenges 


READ 180 Classrooms 


According to the Milwaukee’s grant proposal, READ 180 class sizes were to be limited to 25 
students per section. Suggestions from building administrators revealed a preference toward 
assigning students to classes within grade levels, although unmixed grade level sections were not 
a requirement for the project design. 


Recommended Intensity/Dose for Students 


The READ 180 supplemental reading intervention class lasted for 90 minutes each day. 
According to the project design, students would remain in the intervention for a minimum of one 
year and a maximum of two years, although students could exit from the intervention if they 
reached district-approved proficiency requirements in reading. 


Logic Model 


The logic model, depicted in Figure 1, suggests that, in order for change to occur, a number of 
resources are required—highly qualified teachers, READ 180 instructional materials, and 
professional development. One of the largest resources included in this plan was the hiring of 
highly qualified literacy intervention teachers. Eight reading intervention teachers were to be 
hired to teach 90-minute supplemental reading intervention classes in the five participating 
schools per day with 15—21 students in each block. The program also was designed to involve 
several other stakeholders (as additional resources) who have experience in adolescent literacy 
education and language and literacy development. These other stakeholders were the District 
Identified for Improvement (DIF) supervisors, literacy coaches, and other experienced READ 
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180 intervention teachers. As the following sections describe, these individuals would provide 
coaching support to the newly hired intervention teachers as they begin implementing the 
program. District and building administrators would oversee the ordering and delivery of the 
required instructional materials and would support and supervise the instructional practices of the 
intervention teachers. For example, READ 180 requires a 90-minute time block for effective 
implementation and computer accessibility. Therefore, building administrators must ensure that 
the school scheduling allows for 90-minute blocks and that the classrooms have the needed 
technological resources. 


Professional Development Model 


Four types of professional development opportunities were intended to serve as another set of 
resources for implementing the program. In conjunction with Scholastic (READ 180) and Carroll 
University, the Wisconsin Department of Public Instruction and MPS prepared to provide 
training and professional development to develop highly qualified teachers and administrators 
who would have the skills needed to support the implementation of the READ 180 intervention 
program for struggling readers in Grades 6-10 in participating schools. Scholastic, Carroll 
University, the Wisconsin Department of Public Instruction, and MPS sought to deliver 21st 
century professional development about how to effectively implement the READ 180 program 
that is consistent with the principles of adult learning; utilizes technology; and provides 
opportunities for interaction, collaboration, professional reading, practice, reflection, feedback, 
data analysis, and problem solving. This subsection describes the professional development 
model designed for groups of READ 180 teachers, for building administrators, and for ongoing 
instructional support to individual teachers. 
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Figure 1. Logic Model, Year 1 Plans 


seroma | comggcieem  ec 
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_ Intervention Teachers Implement 
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Implement RED Routines for / 
Student Engagement ; 


{@.g., teaching vocabulary, oral 
cloze, think (write)-pair-share, 
idea wave, etc.) 
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Group Professional Development Model for READ 180 Teachers 


Scholastic provides start-up and READ 180 implementation training, including training on 
strategies to differentiate instruction for students with disabilities and English language learners; 
use of data to assess student achievement and inform instruction; traits-based writing strategies; 
classroom management and design; and software management instruction. In Year 1, new READ 
180 intervention teachers were required to participate in three days of READ 180 start-up 
trainings. Days 1 and 2 would introduce teachers to program components and demonstrate how it 
addresses individual needs through small-group instruction, computer software, and high interest 
literature. Day 3 would expand teachers’ knowledge; review classroom management and goal 
setting; provide time to learn advanced features of the management system; and focus on using 
reports to differentiate instruction. Attendance of experienced READ 180 intervention teachers at 
these trainings was optional. 


In addition, the model called for intensive ongoing professional development sessions that would 
provide specific curriculum implementation training as well as best practices in literacy and data 
analysis. The ongoing training would involve teacher participation in Scholastic online RED 
courses. These courses were designed to increase teacher knowledge about adolescent literacy. 
RED course lessons guide the teacher through practice and real-world application. In Year 1, the 
online course topics included “Best Practices for Reading Intervention” (for novice READ 180 
teachers) and “Strategies for Teaching Striving Readers” (for all READ 180 teachers). 
Participating in the RED courses was required. 


Teachers also were required to participate in monthly roundtable sessions over the course of the 
program. During these sessions, teachers could discuss what they learned in the RED courses, 
such as strategies for delivering whole-group instruction in phonemic awareness, phonics, 
vocabulary, fluency, and comprehension. These two-hour sessions also were to provide teachers 
an opportunity to ask questions and share ideas with one another. 


The program aimed to ensure that reading intervention teachers were highly qualified. Carroll 
University offered graduate-level coursework, which was required for teachers without a reading 
license. Teachers could earn credits toward their reading license and teachers who already had a 
reading license were to be granted credits toward their reading specialist license. Tuition 
reimbursement of up to 12 credits was offered to teachers who earned a B or better. 


Professional Development Model for Building Administrators 


Building administrators were required to attend a half-day orientation to the READ 180 
classroom model in Year 1. During the orientation, administrators would receive a general 
overview of READ 180 and would personally experience the READ 180 classroom rotations— 
small-group instruction, instructional software, and modeled and independent reading—to better 
understand the student and teacher experience in the program. 
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Ongoing Instructional Support for Individual READ 180 Teachers 


Another form of ongoing professional development provided to the intervention teachers was to 
provide mentoring from more experienced teachers and coaches. The professional development 
model indicated that a coach from Scholastic would conduct observations of each READ 180 
intervention teacher once per month and provide feedback. The project coordinator from MPS 
also would conduct observations and provide feedback as needed. In addition, DIFI supervisors, 
literacy coaches, or experienced READ 180 teachers would spend time in the classroom with the 
teacher observing, providing feedback, and modeling appropriate instructional strategies as 
needed. 


Planned Classroom Instruction Model 


The previously described professional development opportunities, combined with the 
instructional materials and logistical building support, were designed to enable teachers to 
implement the READ 180 model with fidelity. 


Recommended Use of Instructional Time in Each Class 


Under the READ 180 classroom model, each day the teacher is to present systematic and explicit 
instruction in essential reading, writing, and vocabulary skills and strategies designed to move 
students from teacher-led instruction and modeling to guided and scaffolded practice and then 
independent practice in each class. Concretely, the 90-minute class involves students in a variety 
of instructional routines that the students participate in daily, beginning with whole-group 
instruction and skill lessons in phonemic awareness, phonics, fluency, vocabulary, text 
comprehension, writing, and grammar during the first 20 minutes of each period. For the next 60 
minutes, students rotate among three activities: small-group instruction, modeled and 
independent reading, and instructional software on computer workstations. During this time, 
students receive intensive intervention that builds on the whole-group instruction. 


During small-group instruction the teacher provides direct instruction to build students’ reading, 
vocabulary, and writing skills using the READ 180 rBook and other instructional materials. This 
time is designed to customize instruction for the individual needs of the students. The teacher 
also may use this time to confer with individual students or check in on the other rotations when 
the students in the small-group rotation are working independently. 


The modeled and independent reading rotation is designed to provide students with an in-class 
opportunity to listen to modeled reading of grade-level literature from the READ 180 
Audiobooks or to read independently from the READ 180 Paperbacks. During this time, students 
also are expected to respond to the reading in writing using a writing log or journal. 


The computer workstations are equipped with individualized instructional topic software that 
provides each student with ability-based instruction, all the while assessing the student’s 
progress. The software is designed to engage students in instructional activities in four “Learning 
Zones”: Reading Zone, Word Zone, Spelling Zone, and Success Zone. Progress reports are 
created for each student, which the teacher can review and use to differentiate instruction. The 
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last 10 minutes of each period is a wrap-up session at which the teacher facilitates discussion 
about the day’s lesson. 


Two additional components of the READ 180 model are essential instructional strategies that the 
teacher must use throughout the day. The first, referred to as RED routines, are designed to 
engage students in the instructional activities. The second component is the use of assessment 
data to guide instructional decisions. As mentioned previously, the teacher must use the progress 
reports created by the READ 180 software system along with other data sources to identify the 
individual strengths and weaknesses of students and then tailor the difficulty of the text, pacing, 
or skill development to meet their individual needs. 


Recommended Grouping of Students for Instruction 


The program allows for flexible grouping of students. READ 180 teachers may create 
heterogeneous-ability groups or groups based on other factors such as student behavior. 


Targeted Areas of Reading/Targeted Learning Strategies 


READ 180 provides individualized, adjusted instruction in the five essential elements of reading 
instruction that were identified by the National Reading Panel (2000): The teacher is equipped 
with a curriculum for teaching essential reading skills—phonemic awareness, phonics, fluency, 
vocabulary, text comprehension, and writing and grammar. 


The READ 180 instructional model promotes a balance of teacher-led direct instruction and 
computerized differentiated instruction every day, and it encourages differentiated and flexible 
instruction. The 90-minute lesson cycle begins and ends with teacher-led whole-group 
instruction. The whole-group instruction begins with the teacher providing systematic instruction 
of specific skills in reading, writing, and vocabulary to the whole class. During the whole-group 
instruction, students are provided the opportunity to use structured Red Routines such as Oral 
Cloze or Think (Write)-Pair-Share to practice speaking, listening, reading, and writing. 


Students then divide into three smaller groups and rotate among three stations. Daily small-group 
direct instruction activities allow the teacher to monitor and address each student’s individual 
needs. The instruction is used for students who are having difficulty with a particular skill and 
provides intensive instruction or conferencing with individual students. READ 180 Topic 
Software provides one-on-one dynamic instruction as it continuously assesses and adjusts to 
students’ individual needs and learning rates. The Structured Independent/Modeled Reading 
provides students with access to a variety of literature and holds them accountable for their own 
reading. 


Reading/Text Materials Provided to Students/Used as Part of the Intervention 


The program design provided students, teachers, and classrooms with the following instructional 
materials: 


e Interactive student work texts called rBooks (one for each student) 
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e READ 180 Paperback and Audiobooks (classroom set) 
e READ 180 instructional software access (for each student) 


e Instructional technology (e.g., computers, headphones, tape recorders—for each 
classroom) 


e Resources for Differentiated Instruction (RDI books), which include lesson plan passages 
and graphic organizers designed to provide targeted skills instruction (for each teacher) 


e READ 180 anchor videos (for each teacher) 
Intended Uses of Technology as Part of the Intervention 


During each class period, students would spend approximately 20 minutes at the computer 
workstation rotation. The topic software at the computer station has three “Zones”: the Reading 
Zone, Word Zone, and Spelling Zone. The software provides continuous, scaffolded practice and 
opportunities for repeated reading of leveled text. 


Technology also was intended to be incorporated into instructional activities during the 
structured, independent reading rotation. During this time, students could listen to audiobooks 
with age-appropriate text designed to motivate students to read. 


Intended Student Assessment Procedures 


READ 180 utilizes the Scholastic Reading Inventory (SRI), a computer-adaptive assessment to 
measure student performance and report scores in lexiles. Lexile measures are based on word 
difficulty and sentence length or semantic difficulty and syntactic complexity. The SRI was to be 
administered initially as a screening measure to determine baseline reading level and then three 
additional times throughout the year to monitor student growth. 


According to the program design, student performance also was to be tracked by READ 180 
Topic Software, rSkills Tests, Scholastic Reading Counts! Quizzes, and READ 180 rBook 
multiyear assessments of skills in phonemic awareness, phonics, fluency, comprehension, 
vocabulary, spelling, and writing and grammar to inform instructional decisions for 
differentiation of instruction to the needs of the individual student. In addition to the formative 
and summative assessments used in the READ 180 program, MPS used the Northwest 
Evaluation Association (NWEA) Measures of Academic Progress (MAP) benchmark 
assessment. The NWEA MAP is administered three times a year districtwide in Grades 3 
through 10. 


Planned Experiences for Control Students During the Intervention Period 


According to the study design, both treatment and control students would receive regular English 
language arts instruction in the same classrooms from the same teachers. Treatment students 
would in addition receive a supplemental literacy intervention using the READ 180 program. 
Control group students would not participate in any supplemental instruction. Instead, these 
students would be enrolled either in study hall or in an elective. 
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Part II: Implementation Study 


The evaluation will address the following research questions related to program implementation: 


Research Question 2: With what fidelity did the program implement the professional 
development model and what factors mediated the level of implementation? 


a. To what extent were professional development opportunities provided and attended as 
planned? 


b. How effective were professional development opportunities in preparing teachers to 
implement READ 180 and appropriate literacy instructional content and strategies? (In 
other words, what was the perceived quality of the professional development and the 
perceived impact on teachers’ behaviors? Do teachers feel prepared and comfortable 
implementing the READ 180 classroom model?) 


c. To what extent were the identified curricular materials and resources made available to 
intervention teachers as planned? 


Research Question 3: With what fidelity to the model did classroom intervention teachers 
implement READ 180 and what factors mediated the level of implementation? 


a. To what extent did intervention teachers implement the READ 180 classroom model with 
fidelity? 


b. Did the students assigned to the treatment group attend the READ 180 class as planned? 


The evaluation used a systematic approach to determining the extent of fidelity to both the 
professional development model and the classroom model. The following sections provide an 
overview of the data sources and rubrics that were used to calculate a fidelity score for each 
teacher (which will be summarized at the school level for the purposes of reporting).* Although 
systematic in nature, these scoring systems are primarily being used to aggregate across a large 
number of data sources and do not necessarily represent a true quantitative measure of program 
fidelity. A weighting procedure is used to ensure that certain aspects of fidelity or certain data 
sources do not contribute too much or too little to the determination of overall fidelity. The 
fidelity scores will be reported simply as high, medium, or low implementation in accordance 
with the requirements of the grant and the stakeholders. 


In addition to reporting implementation fidelity scores, this section also provides detailed 
summaries about particular aspects of program implementation. 


Implementation Data Sources and Collection Methodology 


Several data sources were used to measure implementation of the professional development 
model and classroom model. 


“Implementation measures will not be reported as attributed to a specific school so as not to identify individual 
teachers. We will report only the number of schools achieving a particular level of implementation (i.e., high, 
medium, or low). 
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Professional Development Logs 


The project coordinator from MPS was asked to maintain professional development logs to 
indicate the nature of professional development opportunities provided, when they occurred, and 
who participated. The logs will include records of the following professional development 
opportunities: 


e READ 180 orientation trainings 

e Building administrator professional development trainings 
e Scholastic RED courses 

e Monthly roundtable sessions 


e Graduate-level coursework, including the following courses: Emergent Literacy, Issues in 
Children’s and Adolescent Literature, Reading Assessment in the K-12 Classroom, and 
Practicum in Assessing and Teaching Reading (as needed) 


The project coordinator also was asked to coordinate with Scholastic to maintain coaching logs 
to document the frequency of coaching sessions between Scholastic and each READ 180 
intervention teacher.’ At the end of the school year, AIR reviewed and compared these records to 
original professional development plans to respond to Research Question 2. 


Interviews With READ 180 Teachers and Building Administrators 


AIR conducted structured interviews with READ 180 teachers and building administrators to 
assess implementation of the professional development model and classroom model. The 
research team developed interview protocols designed to capture perceptions about professional 
development quality, implementation of the READ 180 model, and facilitators and barriers to 
implementing the program. Teachers were interviewed in the fall and spring, and administrators 
were interviewed in the spring and summer. Respondents were asked whether they found the 
professional development sessions helpful and whether they modified their practices because of 
what they learned during the professional development sessions. Interviews also were used to 
gather data from teachers and administrators on the extent to which they had access to READ 
180 and other instructional materials, such as technology equipment, and any barriers to using 
the provided resources. 


To facilitate determining the extent of implementation of the classroom model from teacher and 
principal interview responses, the evaluation team developed an interview scoring rubric. The 
rubric maps teacher and principal responses to a rating of high, medium, or low implementation 
and is provided in full detail in Appendix A. The interview rating rubric also demonstrates how 
interview data were incorporated into the professional development model fidelity score. 


° The contract between MPS and Scholastic. was not finalized during the school year and therefore coaching 
sessions did not occur as planned. As a result, no coaching logs were available. 
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Classroom Observations 


Each of the eight intervention teachers were observed four times during the year for a total of 32 
classroom observations.° Observations were staggered across the school year in order to assess 
change in implementation over time. The observation protocol (provided in Appendix B) focuses 
on key characteristics of the READ 180 classroom model and teacher instructional practices. 
Classroom instruction was rated as high, medium, or low in implementation on the following 
aspects of the intervention model: whole-group instruction, small-group instruction, modeled and 
independent reading, computer rotation, and whole-group wrap-up. These ratings were then 
combined into an overall score for each observation (as shown in Appendix B) and contributed 
to the overall classroom fidelity score as described later in this section. 


Extant Data 


The Scholastic Achievement Manager (SAM) provides information about the amount of time 
students spend using the READ 180 computer software. The SAM system is a database that logs 
how students interact with the READ 180 software, how often they use the software (i.e., 
number of days per week), and for what duration they use the software (i.e., number of minutes 
per session). MPS had a system in place for measuring the extent of classroom-level 
implementation that is based on SAM data (summarized as high, medium, or low 
implementation). These MPS-calculated SAM scores were used in our fidelity rating system as 
shown later in this design summary. In addition to SAM data, period attendance records for the 
READ 180 90-minute block were used to describe the dosage of the intervention for students in 
the treatment group. Furthermore, classroom enrollment data and attendance data for treatment 
and control students were collected and examined to ensure that control students were not 
enrolled in a supplemental reading class. 


Implementation Analysis Methodology 


Implementation data were analyzed for two purposes. First, implementation data were reviewed 
and scored according to a priori rubric rating levels to tabulate implementation fidelity scores. 
Scores were assigned at the teacher/classroom level, and there are separate fidelity scores for the 
professional development model and the classroom model. Second, implementation data were 
analyzed to provide contextual and descriptive information about implementation of the 
program. The following section presents the analytic methodology used for each purpose. 


Plan for Measuring Adequacy and Fidelity of Implementation 
Professional Development Model Fidelity Scores 
AIR developed a rubric for assigning an implementation fidelity score related to professional 


development and instructional supports. The rubric specifies indicators related to the quantity 
(i.e., number of sessions attended) and quality (i.e., perceived value added) across a 


°Classroom, rather than teacher, was our unit of analysis, because one teacher was replaced during the year and 
another teacher had a long-term substitute during the time of one observation. 
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measurement scale for the instructional supports and professional development opportunities. 
Scores were assigned for each item in Table 3 using the scoring rubric shown in Appendix C. 
The rubric specifies the primary data source to measure each item, and some items have multiple 
data sources that address the topic. In general, professional development quantity determinations 
were based on the professional development logs, and quality determinations were based on the 
teacher and principal interviews. Furthermore, a combination of interviews and classroom 
observations were used for estimates of fidelity related to the quantity of instructional supports 
(e.g., classroom materials). This score was calculated at the teacher/classroom level. 


Table 3. Professional Development and Instructional Supports Fidelity Scoring Rubric 


Program fidelity score for teacher/classroom X 


Items Quantity Score | Quality Score Total Item Score 
READ 180 orientation training Range = 1-3 Range = 1-3 Range = 2-6 
READ 180 roundtables Range = 1-3 Range = 1-3 Range = 2-6 
READ 180 RED course online Range = 1-3 Range = 1-3 Range = 2-6 
Seite READ tay eens es aaa See Ss Ronee se 
SAM data training Range = 1-3 Range = 1-3 Range = 2-6 
Graduate courses Range = 1-3 Range = 1-3 Range = 2-6 
READ 180 conference (Year 2 only) Range = 1-3 Range = 1-3 Range = 2-6 
eas available/need additional Range = 1-3 Range = 1-3 
Quantity of rBooks Range = 1-3 Wf, Range = 1-3 
Quantity of READ 180 Paperbacks Range = 1-3 SILLS Range = 1-3 
Quantity of READ 180 Audiobooks Range = 1-3 Range = 1-3 
Functional computers and equipment Range = 1-3 Wf Range = 1-3 


Y1 range = 17-51 
Y2 range = 19-57 


Classroom Model Fidelity Scores 


As with the examination of professional development fidelity, the study of classroom fidelity 
used multiple data sources (classroom observations, interviews, and extant program data) to 


examine the extent to which the program provided instruction to students in accordance with the 
intervention logic model. In particular, Research Question 3 aims to measure the extent to which 
the intervention teachers are implementing the READ 180 program as intended, and the extent to 
which students attend class and participate in the designed activities. 


AIR developed a rubric to calculate a classroom fidelity score that combined ratings across data 
sources. The rubric specifies indicators related to how closely to the READ 180 model 
instruction was implemented as collected in the data sources outlined earlier. This score was be 
calculated at a teacher/classroom level. 
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The classroom observation protocol provides a rubric for rating components of the READ 180 
model using a 1—3 rating scale. Descriptors for each rating level are shown in the protocol in 
Appendix B. A similar 1-3 rubric was created to rate responses to interview questions related to 
READ 180 classroom instruction (see Appendix A). 


Ratings for student period attendance were assigned using a 1—3 rating scale on the following 
criteria: 


e A rating of 3 indicates that at least 75 percent of treatment students attended for at least 
75 percent of the year. 


e A rating of 2 indicates that 50 to 74 percent of treatment students attended for at least 75 
percent of the year. 


e A rating of 1 indicates that fewer than 50 percent of treatment students attended for at 
least 75 percent of the year. 


Ratings for the SAM data were based on the system that MPS currently has in place for 
measuring the extent of classroom-level implementation (summarized as high, medium, or low 
implementation). A low rating will receive a score of 1, a medium rating a score of 2, and a high 
rating a score of 3. 


Table 4 depicts how the scores across data sources will be combined to establish the overall 
average classroom implementation score for each teacher/classroom. Each data source has a 
scoring system that results in a different number of possible points. A score and item weighting 
system combines the data sources, shown in Table 4. In particular, the score from each 
observation was weighted by a factor of 1.43 to result in an observation score that can range 
from approximately 50 to 150 total points. In contrast, the single score from the SAM and period 
attendance data sources will be weighted by 40 (with a low score corresponding to 40, a medium 
score corresponding to 80, and a high score corresponding to 120). Furthermore, to give equal 
weight to the observations, interview data, SAM data, and period attendance data, each of the 
four observations contributes to the overall score only once (whereas the other scores contribute 
by a weighting factor of five). The overall categorization of classroom implementation was based 
on dividing the scoring range into thirds, with the lowest third corresponding to low 
implementation, the middle third corresponding to medium implementation, and the highest third 
corresponding to high implementation. 
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Table 4. READ 180 Classroom Implementation Fidelity Scoring Rubric 


Item Bae eae anne Weighted Range 

Classroom Observation 1 35-105 x1.43 x1.00 50.1—150.2 
Classroom Observation 2 35-105 x1.43 x1.00 50.1—150.2 
Classroom Observation 3 35-105 x1.43 x1.00 50.1—150.2 
Classroom Observation 4 35-105 x1.43 x1.00 50.1—150.2 
Interview Rating 40-120 x1.00 x5.00 200.0—600.0 
SAM Data Usage Score 1-3 x40.00 5.00 200.0-600.0 
Period Attendance 1-3 x40.00 5.00 200.0-600.0 

Classroom fidelity score for teacher/classroom X 800.0—2400.0 


Additional Implementation Analyses 
Qualitative Interview Coding 


NVivo software for qualitative research was used to support coding and data analysis of teacher 
and administrator interviews. We employed systematic procedures for coding and categorizing 
the data supports exploration and discovery of categorical relationships that derive directly from 
the data and inform the overall schematic structure of qualitative research—both within and 
across sites. An inductive approach to qualitative analysis incorporates systematic methods of 
managing data through reduction, organization, and connection (Dey, 1993; LeCompte, 2000). In 
our data interpretation and reporting process, AIR researchers identified the common themes and 
divergent cases that exemplify characteristics and challenges incurred by participating staff. This 
process encouraged sensitivity to emergent patterns and regularities, along with contrasts and 
irregularities within and across sites (Delamont, 1992). 


Descriptive Summaries of Quantitative Implementation Data 


In addition to using quantitative implementation data (e.g., observations, period attendance) to 
ascertain the fidelity scores according to the a priori rubrics, AIR researchers also tabulated 
descriptive summaries of these data. When appropriate, our findings sections present descriptive 
statistics of central tendency and variation to provide a more robust depict of the extent of 
program implementation and the extent to which students received the planned intervention. 


Professional Development Implementation Findings 


Overall, the professional development model was implemented at a medium level of fidelity. As 
described previously, the professional development fidelity score included a range of 17-51 
possible points, using data from multiple sources. As shown in Table 5, five of eight classrooms 
received a score in the medium range and three classrooms received a score in the high range. 
The average score across all classrooms was 39 (at the top end of the medium range). 
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Table 5. Fidelity of Implementation of the Professional Development 
Model Across Classrooms 


A closer look at fidelity scores on particular professional development opportunities and 


Classroom on are ah Total Score 
Classroom 1 xX 43 
Classroom 2 xX AO 
Classroom 3 X 37 
ee a : 7m 
Classroom 5 X Al 
Classroom 6 xX 37 
Classroom 7 xX AO 
Classroom 8 X 3A 
ae. oe 0 5 3 Average = 39 


instructional supports provides insight into why the level of implementation of the professional 
development model fell within the mid range. Some elements of the professional development 
model received more ratings in the low range than did other items. For example, as shown in 
Table 6, all classrooms received high ratings for the orientation training and all classrooms 


received low ratings on the mentoring support. 


Table 6. Classroom Score Range for Professional Development and Related Supports 


Professional Development and Related Supports 


Number of Classrooms 


(N = 8) 


Low 


Medium 


High 


READ 180 orientation training 


0 


8 


READ 180 roundtables 
READ 180 RED course online 


Mentoring support from Scholastic/READ 180 representative 


SAM data training 


Graduate courses 


Materials available/need additional materials 


Quantity of rBooks 


Quantity of READ 180 Paperbacks 


Quantity of READ 180 Audiobooks 


Functional computers and equipment 


CSINIJSO;LO!Wwlo!;}o;ama;iNnirR|so 


AIPRIRILOININI FPILOINISA 


NIN] BR] O;WIDIN]| OO] BRI] Ww 


Key for items on 2-6 scale: L/Low range = 2, 3; M/Mid range = 4, H/High range = 5, 6 


Key for items on 1-3 scale: L/Low range = 1; M/Mid range = 2; H/High range = 3 
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The sections that follow provide additional descriptive information on implementation of the 
professional development model, which includes access to instructional materials. 


Implementation of Group Professional Development Model for READ 180 Teachers 


As previously mentioned, the original study design called for four types of group professional 
development opportunities for reading intervention teachers: 


e READ 180 orientation trainings and conferences 
e Scholastic RED courses 
e Monthly roundtable sessions 


e Graduate-level coursework 
READ 180 Orientation Trainings 


By study design, new READ 180 teachers were required to participate in three days of READ 
180 orientation trainings. Days 1 and 2 were to introduce teachers to program components and 
demonstrate how READ 180 addresses individual needs through small-group instruction, 
computer software, and high-interest literature. Day 3 was designed to expand teachers’ 
knowledge, review classroom management and goal setting, provide time to learn advanced 
features of the management system, and focus on using reports to differentiate instruction. 
Experienced READ 180 intervention teachers had the option to attend these trainings or not. 
Interviews with teachers indicated that READ 180 start-up trainings were implemented as 
intended though some additional topics were also covered during these sessions. For instance, 
the first day of the trainings focused not only on an introduction to the READ 180 program 
components but also on 21st century assessment theory. On the third day of training, some time 
was spent on grade-book training (which is not READ 180 content, but training by the district 
that focused on entering grades in a systemwide database aligned with standards). Scholastic 
Achievement Monitor (SAM) data training was covered on day 3. 


Attendance at the READ 180 start-up trainings was high. All teachers attended at least one of the 
three days, and all three new teachers attended day 2, set aside solely for new teachers. Although 
experienced teachers had the option of attending or not, most still attended day 1 (3 of 5) and all 
attended day 3. In interviews, teachers reported that these trainings were effective in reviewing 
the READ 180 program and introducing reports in SAM and the grading of students. As one 
teacher summarized, 


[The READ 180 Start up Trainings] were all day for three days.... we focused on 
how to go about [implementing] the first three weeks [of the READ 180 
program]. We covered some of the reports in SAM, like how to log the kids into 
SAM and get our classes registered.... we worked on grading, so the ESIS Grade 
Book...[and] we talked about twenty-first century reading and grading. 
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Online RED Courses 


Fall and spring RED Courses contained separate content and separate requirements for 
participation—in the fall, experienced teachers were not required to take the RED online courses, 
but in the spring all reading intervention teachers were required to take RED online courses. As 
shown in Table 7, only three teachers completed 75 percent or more of required lessons 
(combining the fall and spring). Although completion of courses was inconsistent (three teachers 
took less than 50 percent of courses), overall reading intervention teachers reported enjoying 
their courses and felt they were helpful in reminding them of instructional strategies as well as 
previous knowledge learned from teacher preparation courses. In fall interviews, two reading 
intervention teachers had not yet heard of RED online courses. By the spring interviews, 
however, all teachers had heard of online RED courses and all but two reported participating. 


Table 7. Percentage of RED Online Courses Completed 
(Fall and Spring), PD Logs 


Percentage Completed eae 
Completed 75 percent or more of required sessions 3 
Completed 50—74 percent of required sessions 2 
Completed less than 50 percent of required sessions 3 


Teacher Roundtables 


The eight READ 180 teachers hired for the Striving Readers grant attended teacher roundtable 
sessions with other READ 180 teachers in the district. Professional development logs indicated 
that only six of the nine originally planned monthly roundtables were offered, which lowered the 
fidelity score. As shown in Table 8, typically, most teachers attended the offered roundtables. 


Table 8. Teacher Roundtable Attendance, PD Logs, N = 8 


ee READ 180 Classroom Total Number of 
ilzxilsalalselelwvisa Classrooms Represented 

September 22 and 23 X|X{}|X)xX]xX}|xX} x] x 8 
October 12 and 20 X|X{|X)xX] xX xX | X 7 
November 13 CANCELED 

December 14 and 16 X|X{}|X)xX]xX}|xX} x] x 8 
January 26 and 27 CANCELED 

February 15 and 16 CANCELED 

March 16 and 17 X}|X/|X)]xX}]X}_ xX] xX 

April 12 and 13 X{|xX|xX|]xX/]xX/xX|]xk|xX 

May 17 and 19 x Xx x xX |X 
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In interviews, all READ 180 teachers reported actively participating in the roundtables they 
attended, and generally they did not miss any except for medical reasons. Teachers also reported 
taking a lot of information away from the roundtables, and they especially appreciated the 
opportunity to share ideas with and ask questions of their colleagues. It should be noted that the 
May 17 and 19 sessions were held after the announcement that the Striving Readers program 
would no longer continue, which may have contributed to the lower attendance (5 of 8 teachers). 


Graduate Courses 


The original study design called for teachers without a reading license to take graduate courses to 
obtain their license through Carroll University. Six READ 180 teachers already held their 
reading license and therefore were not required to take the courses. The other teachers did not, 
and Milwaukee Public Schools had no evidence that these teachers completed graduate courses 
offered through Carroll University. In interviews, three teachers reported being enrolled in 
courses at Alverno, Concordia, and Marquette University, but courses at those universities were 
not paid for by the grant and there was no record of course completion. 


Teacher Feedback on Group Professional Development 


Although teachers were generally satisfied with the professional development they received, 
when asked during interviews, READ 180 teachers requested additional professional 
development through additional online RED courses, roundtables, SAM training, and SMART 
Board Training. Reading intervention teachers also suggested some changes to professional 
development, including providing opportunities to observe other READ 180 teachers, better 
differentiating professional development for experienced and inexperienced teachers, and having 
a greater range of topics from which teachers could pick. For example, one teacher explained 
how previous experience observing READ 180 teachers was especially helpful: 


At the end of last year when I was first approached about teaching READ 180 I 
was sent to a school to observe a teacher and that was kind of helpful. It gave me 
an idea of how [READ 180 is] taught.... [The program] should have READ 180 
teachers go to other schools and see other teachers and how they work in their 
classrooms. That might have been helpful. [As a READ 180 teacher, I] kind of 
feel like an island. I’m the only one here that teaches [READ 180]. So I don’t 
really have anybody to go to about this program. 


Implementation of Instructional Support for READ 180 Teachers 


The original study design included instructional support in the form of mentoring of READ 180 
teachers from more experienced teachers and coaches: a coach from Scholastic would observe 
once per month, the project coordinator from the district would give feedback to teachers as 
needed, and District Identified for Improvement (DIFI) supervisors, literacy coaches, or 
experienced READ 180 teachers also could provide feedback as needed. Teachers also were to 
receive instructional materials in a timely manner from the district or their building 
administrator, and classrooms and students were to be appropriately scheduled. 
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Mentoring 


With the exception of one teacher who reported having an experienced READ 180 teacher 
observe her classroom, teachers did not receive mentoring in Year 1. In interviews, the vast 
majority of teachers reported that they did not have any in-school support from a literacy coach 
or other specialist for Scholastic. Nearly all respondents spoke highly of experiences when 
Scholastic mentors came to their school in previous years, though it was unclear from interviews 
whether visits had occurred this year. As one teacher explained, “ Every year I’ll have a visit 
from [Scholastic mentors]. And I think the world of [them]. [They are] very helpful. Very 
constructive in criticisms.... It’s usually spot on.” No logs of mentoring sessions were available. 
It is understood that mentoring sessions did not occur as planned because the contract between 
Scholastic and the district was not finalized. 


District and State Support 


During interviews, READ 180 teachers indicated that district support consisted primarily of 
addressing classroom needs through e-mail; often this entailed requests for materials. The 
majority of teachers (5 of 8) reported that they were receiving frequent district-level support, and 
the majority of reading intervention teachers (4 of 7) felt the district had affected their instruction 
to a large extent. As one teacher explained, “Well, our supervisor has been wonderful at 
providing support, and making sure that she responds to our e-mails and gives us the materials 
we need.” Those who reported lower levels of district support explained that the district often has 
other obligations. Some teachers elaborated that they do not need much support and very rarely 
send e-mail questions. Those in more frequent contact reported e-mailing and receiving prompt 
responses from the district. 


Instructional Materials 


In interviews, several teachers reported issues with receiving adequate amounts of READ 180 
materials. For the majority of teachers (5 of 8), there were delays in receiving materials. The 
most commonly missing items were headphones (3 teachers) and CD players (2 teachers). Other 
items missing were furniture, action books, DVD players, speakers, and computers. In the fall, 
nearly all teachers (6 of 8) felt that the number of READ 180 paperback books were insufficient 
at one or more levels. By the spring, however, nearly all teachers felt that they had a sufficient 
number of READ 180 paperbacks at various levels. Those teachers who commented on an 
inadequate number of paperbacks explained that students in the upper levels needed more books, 
and some of the topics do not capture the student’s interests. Four teachers requested additional 
materials such as articles and periodicals, a projector or document camera, a larger variety of 
books, a SMART Board, and headsets. 


Implementation of Group Professional Development Model for Building Administrators 
Overall, the implementation of the group professional development model for building 
administrators was relatively low, with less than half the principals participating. According to 


the study design, the building administrator was required to attend a half-day orientation to the 
READ 180 model in Year 1.Two of the five principals participated with the READ 180 
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orientation workshop. According to principal interviews, two principals who did not participate 
this year had attended in previous years. None of the principals participated in any additional 
professional development offered by READ 180 this year, though all reported an awareness of 
READ 180 trainings and described their teacher’s participation in professional development as 
medium to high. 


Other Implementation Concerns 


The vast majority of teachers and principals reported issues with scheduling. Nearly all (4 of 5) 
principals reported challenges with student schedules. The challenge came from difficulty 
scheduling 90 minutes for students while still meeting other course requirements. As one 
principal stated, 


That was probably the biggest challenge I saw with the program and if there were 
any drawbacks, we have within the school for special-education students a 
language program and that lasts for 90 minutes, then we have the READ 180, 
which also lasts for 90 minutes. So, that was probably the biggest programming, 
scheduling struggle, to make sure the kids got all of their educational needs met. 


READ 180 Classroom Model Implementation Findings 


The previously discussed, professional development opportunities, combined with the 
instructional materials and instructional support, are designed to enable teachers to implement 
the READ 180 program with fidelity. The present section will address the implementation of the 
READ 180 classroom model, as well as other critical issues to the classroom model, including 
the actual class size, intensity of the intervention students received (i.e., did students receive the 
amount of intervention the classroom model is designed to give?), and other critical issues, such 
as the teachers’ use of RED routines and assessment data to guide instructional decisions. 


The average classroom implementation fidelity score across the eight classrooms was 1,490, a 
score in the medium range. As shown in Table 9, seven of eight classrooms received a rating in 


the medium range and one classroom received a score in the low range. 


Table 9. Overall Classroom Fidelity Scores Across Classrooms 


Cieoan Low Medium High Classroom 
(800—1,333) | (1,334—-1,867) | (1,868—2,400) Fidelity Score 

Classroom 1 xX 1,247 
Classroom 2 xX 1,545 
Classroom 3 xX 1,451 
Classroom 4 xX 1,498 
Classroom 5 xX 1,510 
Classroom 6 xX 1,585 
Classroom 7 xX 1,561 
Classroom 8 xX 1,525 
Total Number of Classrooms 1 7 0 Average = 1,490 
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An examination of ratings on components that make up the classroom fidelity score provides 
additional information about why the level of fidelity in the implementation of the READ 180 
classroom program fell within the medium range. With some exceptions, teachers implemented 
the READ 180 with high fidelity, but low student attendance reduced the overall classroom 
fidelity scores. 


Class Size 


The READ 180 model calls for 15 students per classroom. As seen in Table 10, attendance data 
revealed that on average, reading intervention teachers in the study had 12 students per 
classroom. In interviews, teachers reported having between 4 and 20 students enrolled in their 
READ 180 classrooms. Several teachers indicated that their class size was often low because of 
attendance issues. For instance, the reading intervention teacher who reported having four 
students explained that several students in special education were placed in her class and had to 
be taken out early in the school year. 


The majority of teachers (5 of 8) reported that their class size was appropriate for all students, 
and teachers generally agreed that class sizes are best around 15. Reading intervention teachers 
with fewer than 10 students wanted more students in their sections to better handle the rotations, 
and teachers with closer to 20 students expressed that behavior management was a challenge 
with that many students. One teacher indicated that an accommodation to the READ 180 
classroom model that was due to a small class size—this teacher had individual sessions with 
students and two, instead of three, groups during rotation time. 


Table 10. Class Sizes of READ 180 Classes, 


Attendance Data 
Average Class Size 
Classrooms Across Three 
Sections 
Classroom 1 11 
Classroom 2 12 
Classroom 3 12 
Classroom 4 13 
Classroom 5 7 
Classroom 6 10 
Classroom 7 16 
Classroom 8 16 
Overall 12 
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Student Attendance in READ 180 


The READ 180 model calls for students to receive 90-minute reading blocks five days a week. 
As Table 11 displays, on average across the year and across classrooms, 58 percent of their 
students were present. As shown in Table 11, no READ 180 classrooms received a high level of 
implementation for attendance, and the majority of classrooms (5 of 8) received a low rating, 
indicating that fewer than 50 percent of treatment students in these classrooms attended at least 
75 percent of the year. 


Table 11. Percentage of Students in Attendance 
by Teacher/Classroom, Attendance Data 


Classrooms Prins ee Beata 
Percentage 
Classroom 1 LOW 46.5% 
Classroom 2 LOW 46.6% 
Classroom 3 MEDIUM 73.9% 
Classroom 4 LOW 64.9% 
Classroom 5 LOW 45.2% 
Classroom 6 LOW 57.1% 
Classroom 7 MEDIUM 71.0% 
Classroom 8 MEDIUM 61.9% 
Total 58.4% 


'A high rating indicated that at least 75 percent of treatment 
students attended for at least 75 percent of the year. A medium 
rating indicated that 50 to 74 percent of treatment students 
attended for at least 75 percent of the year, and a low rating 
indicated that fewer than 50 percent of treatment students 
attended for at least 75 percent of the year. 


*Mean classroom attendance was obtained from each student’s 
“total attendance days” divided by “total school days in the 
school year” averaged out by classroom 


READ 180 Classroom Model Components 


Under the READ 180 classroom model, each day the 90-minute class involves a whole-group 
instruction and skills lesson (20 minutes), rotations between small-group instruction (computer, 
small group, and modeled and independent reading rotations each lasting 20 minutes), and the 
last 10 minutes of each period serves as a wrap-up session during which the teacher facilitates 
discussion about the day’s lesson. Overall, the components were implemented with high fidelity. 
As shown in Table 12, in observations across the year, classrooms implemented the small-group 
and computer rotations with the most fidelity: 27 of 32 observations received a high rating; the 
whole-group wrap-up received the lowest ratings across the year, with 16 of 32 observations 
receiving a low rating. 
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Table 12. Classroom Observation Ratings Across Classrooms and 
Observations Rounds 


Number of Observations That 
Received the Rating 
Low Medium High 
(7-11) (12-16) (17-21) 

Whole-group instruction 1 9 22 
Small-group instruction 1 4 27 
Modeled and independent reading 1 6 25 
Computer rotation 2 3 27 
Whole-group wrap-up 16 12 4 
Overall observation score 0 10 22 


Note: Weighted observation component totals ranged from 7 to 21. 


Overall, the vast majority of observations (22 of 32) received a high fidelity rating. As seen in 
Table 13, two classrooms (Classrooms 1 and 7) showed improvement from a medium to high 
rating. 


Table 13. Overall Classroom Fidelity Score by Classroom 
and Observation Time 


Observation Time 
1 2 3 4 
Classroom 1 Medium Medium High High 
Classroom 2 High High High High 
Classroom 3 Medium* Medium Medium Medium 
Classroom 4 High High High High 
Classroom 5 High Medium Medium* High 
Classroom 6 High High High High 
Classroom 7 Medium Medium High High 
Classroom 8 High High High High 


(Low, medium, high ratings were obtained from the classroom fidelity score. Low= 35- 

58, Medium = 59-82, and High =83-105) 

“Different teacher 
Detailed interview and observation data on the components of the classroom model follow. 
Whole Group 
In interviews, all but one teacher reported using a defined approach for their whole-group 


instruction. Observation ratings were relatively high for this component, with more than two 
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thirds (22 of 32) receiving a high rating on whole-group instruction. In interviews, reading 
intervention teachers typically reported spending 20 minutes in whole group and using the Red 
Routines and rBook to guide their whole-group instruction. Some teachers also used their own 
activities. 


Small-Group Instruction 


Observation data also showed relatively high ratings for teachers on this component (27 of 32 
observations received a high rating). In interviews, the majority of teachers described a 
somewhat defined approach to small group. During interviews, reading intervention teachers 
most commonly reported using the rBook to guide their small-group instruction. 


Computer Rotation 


The computer rotation was implemented with high fidelity during most (25 of 32) observations. 
Teachers generally described an intentional approach to computer rotation that was consistent 
and planned according to the READ 180 model. During the fall interviews, however, three 
teachers reported not being able to implement the computer rotation because of technical 
difficulties, but no teacher reported eliminating this rotation in the spring. Technical difficulties 
included general issues with computers and headphones (e.g., the microphones not picking up 
students’ voices) and issues with accessing the internet. 


Although observations demonstrated a high fidelity of implementing the computer rotation 
component, SAM usage data did not. SAM usage data reported the number of weekly minutes 
students spent on the READ 180 instructional software. The minutes are converted into 
implementation ratings that are based on district specifications. Half the classrooms received a 
medium rating for SAM usage, and the other half had a low rating. 


Modeled and Independent Reading Time 


As shown in Table 12, the majority of observations (26 of 32) received a high rating for modeled 
and independent reading time during most observations. In interviews, teachers generally 
described an intentional approach that was somewhat consistent and planned according to the 
READ 180 model for modeled and independent reading time. According to reading intervention 
teachers, the most common strategy teachers used to engage students was the use of reading logs. 
Other strategies were reading quizzes, conferences after finishing a book, reports, and 
worksheets. Two teachers felt strongly that their use of reading quizzes increased the motivation 
of students to read during independent reading time. As one teacher explained, “Since I started 
incorporating those printout tests, it’s gotten them really interested. In fact, it’s gotten to the 
point where I’ve got to slow them down.” 


Whole-Group Wrap-Up 
As seen in Table 14, observation data revealed that the whole-group wrap-up was the lowest 


rated component of the READ 180 model. Half the observations (16 of 32) received a low 
fidelity rating for this component, and only four observations received a high fidelity rating. In 
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interviews, the wrap-up was the eliminated component most commonly reported, and time 
management was the most commonly cited reason for eliminating it. As one teacher explained, 


The wrap-up is really the only one I don’t get done. And that’s my own time 
management issue.... So my 20 minutes at the beginning becomes 25. And I don’t 
want to short the kids. I still want them to have their 20, 20, 20 [minute rotations]. 
I have to short it someplace, so I take it off my 10 minutes at the end. 


In interviews, teachers described varied approaches for whole-group wrap-up. Activities used 
during these times included asking students to answer a question as they exit the class as well as 
passing a ball around the classroom to engage students in answering questions to written 
reflections/exit slips. The most common approach mentioned by five of eight teachers was 
written reflections and exit slips. As shown in Table 14, four classrooms showed improvement in 
implementing the whole-group wrap-up from the fall to the spring. One of the classrooms that 
showed improvement on this component used prizes to motivate students to volunteer during 
whole-group wrap-up. 


Table 14. Whole-Group Wrap-Up Observation Rating 
by Observation Time and Classroom 


Observation Time 
1 2 3 4 

1 Low Low Medium High 

2 Medium Low Low Low 

3 Low Low Low Low 
eae 4 Medium ae Medium eee 

5 Low High Low Medium 

6 Medium Medium High Low 

7 Low Low Medium High 

8 Low Medium Low Medium 


Note. There was a change in teacher for one classroom from the first to second observation. In addition, there was a 
long-term substitute teacher in one classroom during the third observation. Weighted observation totals ranged from 
7 to 21. High = 17-21, medium = 12-16, and low = 7-11. 


Writing Instruction 


The district included an emphasis on writing instruction in their classroom model. All teachers 
reported that writing instruction was incorporated in their daily routine through the self- 
developed activities, worksheets from their rBooks at the end of each workshop, a reading log, 
and two final projects. Only three teachers mentioned the writing component at the end of each 
workshop. Several challenges with the incorporation of writing also were mentioned. Teachers 
reported difficulties working with bilingual students who were unable to write. One teacher 
explained that conferencing is necessary to properly implement writing instruction, and 
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behavioral problems in the classroom can make conferencing difficult. Finally, one teacher 
explained that 90 minutes of a very scripted intervention makes it difficult to incorporate writing. 


Assessment Data 


One essential component of the READ 180 model is the use of data to guide instructional 
decisions. As mentioned previously, the teacher must use the progress reports created by the 
READ 180 software system along with other data sources to identify the individual strengths and 
weaknesses of students and then tailor difficulty of the text, pacing or skill development to meet 
their individual needs. READ 180 provides teachers with a variety of assessment resources to 
inform instruction, including Scholastic Achievement Manager (SAM) Time-on-Task Report, 
rSkills Test Reports, Scholastic Reading Inventory (SRI) Reports, and Reading Counts! Reports. 


All teachers reported using assessment data to inform instruction. As one of the teachers who 
reported using assessment data more frequently stated, “The SAM data is updated all the time. I 
look at it all the time. I look at it to see how they’re doing in different functions.” In interviews, 
principals also reported that teachers are generating data reports and sharing with them, though 
some principals were more familiar with the names and contents of the reports. Principals use the 
data to monitor student progress and have discussions with teachers in staff meetings. By the 
spring interviews, all teachers reported frequently accessing and reviewing READ 180 reports. 
The vast majority of teachers (6 of 8) reported no difficulty generating reports, but some issues 
were identified. For example, one teacher reported that assessment reports are not always 
accurate, when students’ voices are not picked up on computer microphones, for example. One 
teacher explained that there were challenges with accessing reports through the SAM database: 


Just Friday we had issues with SAM not working and I couldn’t print out practice 
quizzes and the Scholastic Reading [Inaudible] didn’t work on the computers. 


Perceived Grade-Level Implementation Differences 


Teachers noted some differences in age of students and their receptiveness to the READ 180 
model, which may have influenced instructional strategies. Some teachers noted that sixth-grade 
students were more willing to try different things, though they were not always as motivated 
because they have more time until high school to get proficient. Eighth- and ninth-grade 
students, in contrast, had a greater urgency and motivation to become proficient, but they can 
find some aspects of the READ 180 program (big writing, the subjects) to be “babyish.” As one 
teacher noted, “With eighth graders, it’s more of an emotional thing where they’re more 
embarrassed to be in READ 180 at the beginning than the other grade levels, and that’s just 
because they’re heading into high school and they want to be treated a little differently.” 


Factors in Fidelity of Implementation 


The present section discusses some facilitators and barriers that may have affected the fidelity of 
implementation. 
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Possible facilitators 


Some possible facilitators in the fidelity of implementation of the READ 180 model were the 
teachers’ prior experience with the program and collaboration with one another. For instance, 
several of the teachers were experienced in READ 180 before being hired for this program. 
Those teachers who had another READ 180 teacher at their school, particularly one experienced 
with the program, reported communicating with them frequently and planning instruction. They 
said that the close proximity of this teacher provided added support. 


Possible barriers 


There were also several barriers to the fidelity of implementation. These were difficult—to- 
manage students, administrator turnover, the absence of school support, tardy receipt of 
materials, and a lack of coaching from Scholastic. 


Several teachers commented that they had students who were especially difficult to manage, and 
two teachers reported that their students were inappropriately placed in classes. These two 
teachers felt their students were functioning at too low a level to handle the READ 180 
requirements. Further, some students were scheduled for the READ 180 class but were actually 
attending other classes. Similarly, four of five principals reported challenges with student 
schedules. Principals reported difficulty finding a 90-minute period to schedule students while 
still meeting the student’s other course requirements. As one principal stated 


[Scheduling] was probably the biggest challenge I saw with the program, and if 
there were any drawbacks, we have within the school for special-ed students a 
language program and that lasts for 90 minutes, then we have the READ 180, 
which also lasts for 90 minutes. So that was probably the biggest programming, 
scheduling struggle, to make sure the kids got all of their educational needs met. 


One administrator was a new principal this year, though he had previous experience as an 
administrator. This may have served as a barrier because new principals often have other duties 
to focus on beyond READ 180. Several teachers from other schools reported struggling with the 
implementation of READ 180 because their principal did not fully understand the program 
model and time requirements and sometimes interfered with student scheduling. 


As mentioned earlier, several teachers did not have their READ 180 materials at the start of the 
school year. Only three of eight teachers received all their materials on time—the tardy materials 
were computers, headphones, and rBooks, all of which are necessary for fidelity of 
implementation. 


Experiences for Control Students During Intervention Period 


This study was designed to test the effect of READ 180 as a supplemental reading intervention. 
Treatment students were to attend their regular ELA class plus READ 180. Control students 
were to attend their regular ELA class plus a nonreading elective or study hall. Table 15 shows 
the percentage of treatment and control students enrolled in READ 180. Only two control 
students (less than 1 percent) ended up enrolled in READ 180, and nearly 70 percent of 
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treatment students were enrolled in READ 180. Thirty percent of treatment students were not 
enrolled in READ 180 by the schools. In many cases, these students were post-random 
assignment principal exclusions. For the purposes of calculating the intent-to-treat effect of 
READ 180, these students are nevertheless considered as treatment students. 


Table 15. Enrollment in READ 180 by Treatment Group 


Enrolled in Not Enrolled in 
Group READ 180 READ 180 Total 
N % N % 
Treatment 233 69.6% 102 30.4% 335 
Control 2 0.7% 282 99.3% 284 


As stated above, control students were to enroll in regular ELA and either a nonreading elective 
or a study hall. AIR worked with MPS to gather data on student enrollment in ELA and reading 
courses. Table 16 shows the pattern of course taking by treatment and control students (the 
average number of courses per semester in four categories). Regular ELA courses are those 
courses students are required to be enrolled in. Other ELA courses were nonreading ELA courses 
that were not considered the main ELA course required of students. Other reading courses 
represent non—READ 180 reading courses in which students were enrolled. 


The table indicates that, as expected, treatment students were mostly enrolled in READ 180, and 
that most treatment and control students had a regular ELA course. The average treatment 
student was enrolled in 0.82 regular ELA courses, as compared with 0.91 for treatment students. 
This difference indicates that treatment students were enrolled in regular ELA courses less often 
than control students, contrary to the design of the study. This result is most likely due to the 
scheduling concerns expressed by principals elsewhere in this report. Control students also were 
more likely to be enrolled in other ELA courses. These additional courses most likely represent 
nonreading elective courses in place of READ 180, which was allowed by the study design. 
Table 16 also indicates that the average control student was enrolled in 0.13 additional reading 
courses. This would seem problematic because control students were not to receive supplemental 
reading instruction, but the average treatment student was enrolled in 0.10 additional reading 
courses, so it seems to balance somewhat across treatment conditions. 


Table 16. Average Number of Courses per Semester 
by Treatment Group 


Regular | Other Other 
Group READ ELA ELA | Reading N 
180 
Courses | Courses | Courses 
Treatment 0.75 0.82 0.46 0.10 335 
Control 0.01 0.91 0.79 0.13 284 
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Ramifications of Implementation Results for Impact Analysis 


The preceding sections describe in detail the fidelity of implementation of the READ 180 
Striving Readers program in five Milwaukee schools. This section provides a brief summary of 
these findings and discusses what these findings might imply for the analysis of program impact 
on student outcomes. Any program, no matter how effective, may not have an effect unless 
participants actually receive the treatment as intended. 


In general, implementation of the READ 180 program was reasonably good. Teachers were 
observed to be implementing the READ 180 model in the classroom with high fidelity. READ 
180 classes tended to follow the designed READ 180 process of whole-part-whole, with students 
going through rotations during the middle period of the class. Although teachers implemented 
the model with fidelity, however, student attendance was generally low. To have the full impact, 
students need to be in class to receive the supplemental instruction. This was often not the case. 
Beyond the classroom, teachers also were to receive professional development and mentoring. 
The overall rating for professional development fidelity was medium—implying that teachers 
did not receive the full level of training and support that was planned. 


From these findings, it seems reasonable to expect that there would be a moderate impact of the 
READ 180 program on student outcomes. Although the model was not implemented perfectly, 
many students did receive READ 180 supplemental instruction. It seems reasonable to infer that 
if student attendance had been higher, the impact of the program would be larger than what was 
observed with this study. 
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Part III: Impact Study 


This study is designed to detect the impact of a two-year READ 180 reading intervention for 
struggling readers in five Milwaukee Public Schools under a Striving Readers grant from the 
U.S. Department of Education. This study will use a strong experimental design to produce a 
rigorous estimate of the impact of the READ 180 intervention on measures of reading 
achievement for struggling students. The evaluation also will explore implementation fidelity 
and the contexts and conditions of implementation that may extend or limit the intervention’s 
effects. The primary purpose is to produce a rigorous estimate of the program’s causal impact. 
The secondary purpose is to assess the fidelity of program implementation in the study schools. 


Because the Striving Readers grant was discontinued, the impact study will focus only on the 
one-year impact of the reading intervention. The initial study was designed to test the effect of a 
two-year READ 180 intervention and was powered on the contingency of being able to pool 
effect estimates across academic years. 


Study Design 


This section will present the key research questions, the sampling plan, data collection plan, and 
the analytic approach to the impact study. 


Research Questions for Impact Study 


The experimental study is aimed at determining the extent to which treatment students receiving 
the supplemental READ 180 intervention improved their reading proficiency relative to control 
students. As initially intended, this study focused on the effect of a two-year READ 180 
intervention for struggling students. With the cancellation of funding for the Striving Readers 
program, however, this report provides analyses of data on a single-year intervention. The 
primary casual research question and four research subquestions are as follows: 


Research Question 1: Does the READ 180 reading intervention improve students’ academic 
performance in reading? 


a. What is the impact of the READ 180 intervention on struggling Grade 6-10 students’ 
academic performance in reading? As a secondary analysis, what is the impact of one 
year of READ 180 participation on struggling Grade 6—9 students’ academic 
performance in reading? 


b. Are there differences in the amount of impact of the READ 180 intervention on students’ 
academic performance in reading between middle school students (Grades 6-8) and high 
school students (Grades 9-10)? 


c. Are there differences in the amount of impact of the READ 180 intervention on students’ 
academic performance in reading between students with identified special needs and 
other struggling readers? 


d. Does the READ 180 reading intervention increase student engagement and self-efficacy 
related to reading? 
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Sampling Plan 


This section describes how students were identified for random assignment and assigned to study 
conditions for Cohort A’ (i.e., the 2010—11 school year). In total, two randomizations occurred in 
which students were assigned to either a treatment condition (participate in their regular ELA 
course plus the READ 180 supplemental literacy intervention) or a control condition (participate 
in their regular ELA course plus attend study hall or a nonreading elective), with the second 
randomization also generating a waitlist of students for the treatment and control groups. 


Sample Selection Process 


For the 2010-11 school year, students in Grades 6—9 across the five study schools were 
considered eligible for random assignment if they met the criteria established in the guidelines 
set forth by Milwaukee Public Schools (MPS) for entrance to the READ 180 program. In 
general, students were considered eligible for random assignment if they were reading at a 
below-proficient level on the basis of the Wisconsin Knowledge and Concepts Examination 
(WKCE), the district benchmark assessment (i.e., ThinkLink), or their teacher-evaluated reading 
proficiency. In addition, some students were excluded from random assignment based on 
predetermined exclusionary criteria for students in special education and English Language 
Learners. Detailed eligibility requirements and exclusionary criteria are described in Part I: 
Intervention and Logic Model. 


MPS research and assessment staff provided AIR with student data on April 29, 2010. MPS 
provided five data files, one for each study school, containing the names and assessment scores 
of all students eligible for random assignment according to the established criteria. These files 
were limited to students enrolled to date in the school for the upcoming year. 


In May 2010, AIR provided the study school principals and programmers lists of eligible 
students and asked them to screen out any students who were ineligible for the study—no longer 
enrolled in the school, were ineligible because of their special education Language! course level, 
or other principal-identified exclusion. After allowing a few weeks for their review and 
providing multiple reminders, AIR received feedback from four of the five schools on the list of 
eligible students. A total of 45 students were deemed ineligible on the basis of a principal request 
(for reasons other than their Language! course level). AIR removed these students from the 
eligibility list as well as other students who did not meet the eligibility criteria of prior test scores 
or English language proficiency level. AIR then used these cleaned versions of the eligibility 
lists for the first round of random assignment. 


For the first round of random assignment, AIR randomly selected students within each study 
school to a treatment condition or a control condition. The first randomization for Cohort A was 
conducted to fill as many of the READ 180 spots as possible (while still maintaining a sufficient 
sample of control students within each grade and school). A total of 504 READ 180 spots were 
available (eight intervention teachers, each teaching three sections per day, with a maximum of 
21 students per class). The first randomization process used a stratified random sample with 


’Cohorts B and C were to be randomized prior to school years 2011-12 and 2012-13. Because funding was 
discontinued, only Cohort A went through the randomization. 
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control sorting (using proc SURVEYSELECT in SAS 9.1) in July 2010. The sample was 
stratified by school and grade and controlled for special education status. The number of students 
receiving treatment in each school was determined by the number of teachers assigned to each 
school and the breakdown of grades within each school. AIR sent the initial lists of treatment and 
control students to the study schools in June 2010. 


On July 30, 2010, MPS provided AIR with an updated school enrollment data file for each 
school. These files contained the names, demographics, and assessment scores of all students 
eligible for random assignment (according to the established criteria) who were enrolled in the 
school for the upcoming year as of July 30, 2010. From this list, AIR identified study students no 
longer enrolled in one of the study schools to determine the number of additional study slots by 
school and grade level available for incoming students who met the eligibility criteria to be 
randomly assigned to the study. 


On August 11, 2010, AIR provided each school principal with another list of eligible students on 
the basis of the updated file. Principals and programmers were asked to identify any students 
they deemed ineligible to participate in the study because of the exclusionary criteria. After 
approximately one week for review and multiple phone and e-mail reminders, only one of the 
five schools provided feedback on the updated eligibility lists. AIR removed these students (52 
students from one school) from the eligibility list as well as other students who did not meet the 
eligibility criteria for prior test scores or English language proficiency level. AIR then used these 
cleaned versions of the eligibility lists for the second round of random assignment. AIR sent the 
updated lists of treatment and control students to the study schools in August 2010. 


The second randomization process was conducted differently than the first randomization. For 
the first randomization, more treatment spots were available than the pool of eligible students 
could fill. The first randomization did not fill all treatment spots for the purpose of maintaining a 
sufficient pool of control students (e.g., if there were 28 eligible students in a grade, 16 might be 
assigned to treatment and 12 to control rather than 21 to treatment and 7 to control). The second 
enrollment file contained 590 eligible students, but there remained 81 treatment spots to fill. 
Because the newly eligible students had not been enrolled in the study schools during the first 
enrollment period (prior to May 2010), these students were suspected to be qualitatively different 
from students in the first randomization (i.e., they enrolled in school later in the summer rather 
than in the spring). Placing 509 students of the 590 eligible students into the control group (as 
would be the case under the sampling plan from the first randomization) would result in an 
imbalance of the combined (across randomizations) treatment and control groups (both in size 
and preexisting achievement). Therefore, the second randomization was conducted by assigning 
each student a random number, sorting students within grade and school, and selecting students 
for treatment on the basis of this random number. If five students were needed for a school and 
grade combination, the first five students on the random list were selected as treatment, and the 
bottom five students were selected as control. This process not only ensured a balanced addition 
of treatment and control students to the sample but also created a waiting list. In addition, the 
final impact model for the experiment included not only stratification variables, but also 
indicator variables that accounted for the different randomization periods (i.e., indicators for 
each randomization period, grade, and school combination). 
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The waiting list created by the second randomization provided the schools with a list of eligible 
students the schools could use to fill vacant spots. To fill vacant spots in READ 180 classrooms, 
principals and programmers informed AIR of students who did not enroll in the school or were 
excluded for other reasons identified by the principal (including reasons that met and did not 
meet the exclusionary criteria). Using the waiting list, AIR staff provided the principal with the 
next student on the school by grade waitlist to the treatment condition as well as the next from 
the bottom student on the school by grade waitlist to the control condition. This wait list process 
continued until the end of September. After this date, the principals and programmers were 
instructed that they could fill any vacant READ 180 spots with any students not identified for the 
control condition in the first or second randomization. Students added by the principal or 
programmers after the randomization periods were not included in the impact study. 


Post-Random Assignment Exclusions 


After the first and second randomization, principals reported multiple reasons students assigned 
to a treatment condition were not placed in READ 180. In many cases, students who were 
randomized into treatment or control conditions did not end up attending the school. These 
students did not know their treatment status and were excluded from the study (129 treatment 
students and 112 control students). During the scheduling process, some students were deemed 
ineligible by principals on the basis of various criteria. Because these students were considered 
nonrandom exclusions, they are included in the intent-to-treat analysis, even though they did not 
enroll in READ 180. 


Total Sample Size 


As Table 17 indicates, 809 students were assigned in the first randomization (434 treatment and 
375 control students). The second randomization placed 78 treatment students and 79 control 
students, and 80 treatment students and 80 control students were selected off the alternate list. A 
total of 591 students were assigned to treatment, and 531 were assigned to control over the two 
randomizations and waitlist process. After removing students who never showed up for school at 
their enrolled school, the final intent-to-treat sample is 462 treatment students and 419 control 
students. Students who did not show up for school are not included in the intent-to-treat sample 
as these students did not attrition with any knowledge of their study status. 


Table 17. Assigned and IIT Sample Size by Randomization 


ee Se Overall 
Study Randomization | Randomization 
Group 1 9 Alternates pene Did Not ITT 
8 Enroll Sample 
Treatment 434 78 80 591° 129 462 
Control 375 79 80 531° 112 419 


“ One treatment and three control students attritioned after Randomization 1 (retained in grade or switched schools) 
and were rerandomized, either during the second randomization, or on the waitlist. 


Table 18 shows the number of control (Ctrl) and treatment (Tx) students by grade and school in 
the baseline sample (expected sample size for the intent-to-treat analysis). 
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Table 18. Baseline Sample Size by School and Grade 


School Grade 6 Grade 7 Grade 8 Grade 9 Total 

Ctrl Tx Ctrl Tx Ctrl Tx Ctrl Tx Ctrl Tx 
Audubon 32 36 35 42 23 37 0 0 90 115 
J-MAC 0 0 0 0 0 25 41 25 41 
Mitchell 19 21 12 14 13 0 0 39 48 
Morse 16 22 25 34 32 21 32 40 105 117 
NWS 18 25 35 32 42 39 65 45 160 141 
Total 85 104 107 122 105 110 122 126 419 462 
Attrition 


Table 19 reports the number and overall attrition rate of students on the basis of the baseline 
randomization (i.e., the number of initial intent-to-treat sample students). The overall attrition 
rate for study students was 30 percent (i.e., students who did not attend school in the district, left 
the district, or otherwise had no outcome data). For the treatment group, the attrition rate was 28 
percent, and for the control group, the attrition rate was 32 percent. The attrition rate for students 
in the treatment condition (78 percent) was greater than the attrition rate for students in the 
control condition (57 percent). This attrition rate reflects the number of baseline students with 
MAP reading assessment data from June 2011. The missing students reflect both those who did 
not enroll in the school or left the school after randomization, as well as any students who did not 
complete the MAP reading assessment in June 2011. 


Table 19. Attrition Rate Between Baseline Sample and 
Reading Proficiency Data 


MAP Reading 
Condition Baseline Assessment, Attrition Rate 
June 2011 
Treatment 462 335 27.5% 
Control 419 284 32.2% 
Total 881 619 29.7% 


Figure 2 is a consort diagram of the number of students assigned to the study and the reason 
students were removed. These numbers reflect students no longer in the study but do not reflect 


students who did not participate in the June 2011 MAP assessment. 
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Figure 2. Consort Diagram Showing Randomization Process and Final Intent-to-Treat 
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Data Collection Plan 


This section will describe the data collected for the two primary outcomes of this study: student 
reading proficiency and student self-efficacy and motivation. 


Student Reading Proficiency 


The Measures of Academic Progress (MAP) assessment served as the primary outcome measure 
of student reading proficiency used for the analytic analysis. Developed by the Northwest 
Evaluation Association, the MAP is a computer-adaptive assessment aligned to state standards 
that provides the school with immediate feedback on student progress.® The scores represent a 
developmental scale and are comparable across grade-level. Beginning in the 2010—11 school 
year, Milwaukee Public Schools identified the MAP as the primary assessment measure of 
student progress, replacing the ThinkLink (which had been identified as the primary outcome of 
interest in the original proposal). The MAP is administered to all MPS students in mathematics 
and reading three times a year: October, February, and June. For the purposes of this study, the 
reading portion on the June MAP assessment will be used as the primary outcome measure for 
student reading proficiency. 


Reading Engagement and Self-Efficacy 


American Institutes for Research developed a student survey to determine whether participation 
in the READ 180 reading intervention had an impact on student engagement and self-efficacy 
related to reading. The survey asked students to respond to items related to self-efficacy, as well 
as the constructs of behavioral engagement, emotional engagement, and cognitive engagement 
with reading. 


All school principals in the participating schools were invited in March 2011 to preview the 
student reading survey and add other questions they would like to include in the analysis. Two 
principals provided additional survey questions. After finalizing the survey, school schedulers 
received copies of student surveys to distribute to all ELA classrooms in April 2011. All Grade 
6-9 students at the participating schools, regardless of study status, were instructed to receive the 
survey. Each survey had a unique ID with an attached name page. Students were asked to write 
their name on the name on the name page and return it to the teacher. Student surveys and names 
were returned in separate mailings, and American Institutes for Research attached student IDs to 
the unique IDs for the analysis. Only surveys of students participating in the study were used for 
the analysis. 


Analytic Approach to the Impact Analysis 
The primary focus of the study design was to estimate the effect of a two-year supplemental 


literacy intervention using READ 180. Because the Striving Readers grant was discontinued, 
however, the study design was revised to estimate the effect of a one-year intervention. This 


8 


http://www.nwea.org/products-services/computer-based-adaptive-assessments/map 
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section will summarize the methods used to measure the impact on students at the end of one 
year. Additional details of the impact analysis methodology are presented in Appendix D. 


Models 


To measure the impact of the READ 180 intervention on reading proficiency for students in the 
five MPS study schools, we will test for both intent-to-treat (ITT) and treatment-on-the-treated 
(TOT) effects. Although we will consider both models in our analysis, the primary model used 
for reporting the effectiveness of the READ 180 intervention will be the ITT analysis. The ITT 
group includes all students randomized to the study who enrolled in the school and grade level at 
which they were randomized in the 2010-11 school year. The basic models for the ITT estimate 
of the impact of the intervention are as follows: 


(1) Yij = Bo + BiTy + ZBISGRy + &ij 
(2) Yij a Bo + Bi Tj + XPijSGRiy + X=BP; at Ejj 
(3) Yy = Bo + BiTy + ZPYSGRy + LPiPi + XPD; + ij 


In this model, Yj; represents the student-level, postintervention outcome (e.g., reading 
achievement on the MAP reading assessment completed in June 2011). Tj; is an indicator of the 
treatment status of student i in school j, Pi represents two pretest standardized test scores (i.e., 
WCKE and ThinkLink), D; represents student-level covariates (e.g., gender, ethnicity, economic 
status, special education status, and ELL status), and SGRj represents a school-by-grade-by- 
randomization block for a given student. In addition, models include missing data indicators for 
prior achievement scores (equaling 1 for students with missing WKCE or ThinkLink scores, 
respectively). Student-level covariates to include in the models will be selected using backward 
selection criteria with a critical value of 0.2 (t-statistic significance) for inclusion. 


The TOT analysis will be conducted to account for the possibility of treatment crossover and 
other forms of treatment noncompliance. The TOT estimate will account for that noncompliance 
through the use of a two-stage least-squares approach, using random assignment as an 
instrumental variable for treatment receipt (Angrist, Imbens, & Rubin, 1996). The second stage 
model for the TOT analysis is 

(4) Yi j- Bo + BiR ii F XPiySGRj + XBiPi + XPD; + &jj 


In this model B; is the estimated impact of READ 180 for a student i who enrolled in READ 180 
(R’) based on their assignment to the treatment group. 


Subgroup Analyses 


In addition to the one-year ITT analysis, analyses were conducted to look at how different 
student subgroups perform the outcome measures. These subgroup analyses were as follows: 


Special Education Status. This subgroup analysis examined the impact of a one-year READ 180 
intervention on students identified prior to randomization for special education. The subgroup 
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analysis will follow the empirical structure for model (3) plus add an additional interaction term 
between treatment status and special education status: 


(5) Yi j~ Bo + Bi Tj + BoT*SPED, + XP jSGRy + X=p,P; + =B,D; a Ejj 


The value of the coefficient 3 will reflect the average treatment effect on special education 
student assigned to treatment. 


English Language Learner (ELL) Status. This subgroup analysis will examine the impact of a 
one-year READ 180 intervention on students identified prior to randomization as an ELL. The 
subgroup analysis will follow the empirical structure for model (3) and add an interaction term 
between treatment status and ELL status: 


(6) Yi j~ Bo F Bi Tj + BoT*ELL, oF XPijSGRy oa XB ;P; aH =B,D; a gjj 


The value of the coefficient B» will reflect the average treatment effect on ELLs assigned to 
treatment. 


Middle School Versus High School Students. This subgroup analysis will examine separately 
the impact of a one-year READ 180 intervention on middle school (Grades 6-8) and high school 
(Grade 9) students. The subgroup analysis will follow the empirical structure for model (3). 


Description of the First-Year Sample 


The following sections provide an overview of the numbers and basic characteristics of schools, 
teachers, classrooms, and students involved in the study. 


Schools 


The characteristics of the five study schools varied in size, as well as in the proportion of the 
student populations receiving special education services, classified as English language learners, 
and receiving free or reduced-price meals. Student populations within the five study schools 
ranged in size from 625 to 1,201 students, with a mean student population of 931 students per 
building. Demographics for each of the five study schools are presented in Table 20. 
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Table 20. Characteristics of Wisconsin (Milwaukee) Striving Readers Schools, 2010-11° 


. % Free or 
0) 1) 
Grade Total Ho 7 Nea Reduced- 
School Name Special Language 3 
Span Enrollment : Priced 
Education Learners 
Meals 
Audubon 6-8 625 20.6% 24.2% 83.4% 
eee 9-12 1,081 21.6% 2.1% 75.6% 
ampus 
Mitchell K-8 730 18.1% 34.8% 94.8% 
Morse Marshall 6-11 1,201 22.2% 3.2% 67.4% 
Northwest Secondary School 6-12 1,019 28.5% 0.5% 81.4% 
Teachers 


MPS hired eight reading intervention teachers for the start of the 2010-11 academic year as 
called for in the original study design. Seven of the eight teachers were to be assigned to a single 
school, with one teacher splitting time between JMAC and Northwest Secondary School, as 
shown in Table 21. 


In December 2010, the full-time reading intervention teacher serving JMAC and Northwest 
Secondary transitioned to a new role at which point a new reading intervention teacher was 
hired, also splitting time between the two schools. Additionally, one reading intervention teacher 
at Morse was replaced by a long-term substitute in November of 2011. 


Table 21. Distribution of READ 180 Teachers by School 


aca cea 
Audubon 2 
James Madison Academic Campus 2/3 
Mitchell 1 
Morse Marshall 2 
Northwest Secondary School 21/3 
Total 8 


* Demographic and enrollment information was obtained from the Wisconsin Department of Public Instruction 
website’s WINNS system: 

http://data.dpi.state. wi.us/data/GroupEnroll.aspx? OrgLevel=st&GraphFile=BlankPageUrl&S4orALL=1&SRegion= 
1&SCounty=47&SAthleticConf=45&SCES A=05&Qquad=demographics.aspx (accessed on December 22, 2011). 
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Of the eight intervention teachers working in the study schools in spring 2011, four of the eight 
indicated that they had prior experience with the READ 180 model. In addition, five of the eight 
instructors reported previous experience with programs that focused on struggling readers, 
including the Direct Instruction reading program, Pebble Creek, and Soar to Success. 


For the majority of the teachers interviewed in fall 2011 (five of nine), it was their first year 
teaching at their respective schools, with the total years of teaching experience ranging from 3 to 
30 years. All but one teacher reported holding a reading license, though two of the eight teachers 
who indicated that they held a reading license in spring 2011 specified that they held an 
emergency reading license. 


Classrooms 


Originally, the study called for each reading intervention teacher to teach three to six 90-minute 
reading blocks daily. As shown in Table 22, in practice, each teacher taught three sections of 
READ 180 during the 2010-11 academic year, with the shared reading intervention teacher 
leading two sections of READ 180 at JMAC and one section at Northwest Secondary. Northwest 
Secondary had the most available sections of READ 180, with seven, and Audubon and Morse 
each had six available sections of READ 180 for treatment students. Mitchell and JMAC had the 
fewest sections, with three and two sections, respectively. 


Table 22. Number of READ 180 Sections by School 


Number of 


School Teacher ‘ 
Sections 


Teacher 1 
Teacher 2 


Audubon 


James Madison Academic 


Teacher 1° 
Campus 


Mitchell Teacher 1 3 
Teacher 1 3 
Morse Marshall 
Teacher 2 3 
Teacher 1 


Northwest Secondary School Teacher 2 
Teacher 3° 


Total 8 24 


“Shared teacher between JMAC and Northwest Secondary, as of December 2010. 


According to the READ 180 program model, all sections were 90 minutes in length and students 
in the treatment group were expected to receive a minimum of 90 minutes of intervention 
instruction daily, or 450 minutes per week. 
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Students 


Within the study group, as shown in Table 23, the characteristics of students in the treatment and 
control conditions were similar in most areas. Overall, as shown in Table 23, the study sample 
consisted of more males than females and the majority of students received free or reduced-price 
meals, an indicator of low socioeconomic status. Slightly more than one third of study students 
received special education services during the 2010-11 academic year and less than 10 percent 
were considered to be English language learners (ELL). In terms of ethnicity, the majority of 
students in the sample were African American (70 percent). 


Table 23. Characteristics of Treatment and Control Students 


enapepnice (Saad | Bet) ea 
(N = 769) 
Special education 37.4% 33.7% 35.5% 
English language learner 7.4% 8.2% 7.8% 
Free or reduced-priced meals 85.9% 89.5% 87.8% 
Female 40.3% 37.2% 38.8% 
African American 71.6% 68.6% 70.1% 
Hispanic 4.0% 2.0% 18.9% 
White 17.5% 20.2% 7.2% 
Other* 0.8% 1.0% 3.9% 


“Other includes students that identified as being of Asian or Native American descent. 


Note: Table reports percentage of students who fall into specified category. Students may be 
classified under multiple categories, and thus percentages do not sum to 100. 


Pearson’s chi-square was used to test the hypothesis that students in intervention and control 
groups were similar for each demographic group. No group was found to be significantly 
different at the 0.05 level. 


Impacts on Teachers and Classroom Practices at the End of One Year 


As introduced in the overview of the intervention and logic model, the READ 180 program was 
designed to provide the participating schools with a series of resources, including instructional 
materials, newly hired literacy intervention teachers, and professional development 
opportunities. In the event that these resources were provided as planned, the logic model 
predicts that the intervention teachers will become knowledgeable about the READ 180 
classroom model as a supplemental literacy class for struggling readers and become comfortable 
implementing it. In addition, READ 180 resources and supports are expected to enhance 
intervention teachers’ knowledge and use of effective instructional practices. Ultimately, the use 
of effective classroom practices should lead to changes in students’ behavior and academic 
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performance. This theory of change is depicted within the simplified logic model in Figure 3. A 
more detailed logic model can be found in Part I (Figure 1) of this report. 


Figure 3. Simplified Logic Model 


Teacher Classroom Student 


Resources ? 
Outcomes Practices Outcomes 


As previously described, four types of professional development opportunities were available to 
reading intervention teachers: 


e READ 180 orientation trainings and conferences 
e Scholastic RED courses 
e Monthly roundtable sessions 


e Graduate-level coursework, including the following courses Emergent Literacy, Issues in 
Children’s and Adolescent Literature, Reading Assessment in the K-12 Classroom, and 
Practicum in Assessing and Teaching Reading (as needed) 


Based on the logic model, reading intervention teachers participating in the professional 
development were expected to see increases in knowledge of the READ 180 classroom model, 
comfort in implementing the READ 180 classroom model, knowledge of instructional practices, 
and comfort in implementing appropriate instructional practices such as student engagement 
activities, progress monitoring through formative assessment, and data-based differentiated 
instruction. 


Measures of Teacher Outcomes 


In order to determine the extent of teacher outcomes related to participation in the READ 180 
program and subsequent professional development, AIR used data from both classroom 
observations and teacher interviews. 


Interview Questions 


Interviews with teachers were conducted in fall 2010 and spring 2011 to gather more detailed 
information about how the intervention affected teachers’ attitudes, knowledge, and instructional 
practices. The research team and literacy staff from MPS developed interview protocols, with 
questions specifically designed to gain targeted information on teacher outcomes, including their 
typical approach to instruction and the READ 180 model and any changes in their approach from 
the beginning of the school year; use of READ 180 reports and data; and modifications to 
teaching practices as a result of participation in professional development. 


In addition, reading intervention teachers were asked to rate the following questions on a scale of 
high, moderate, or low: 
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e How would you rate your familiarity with the READ 180 program model? 


e How would you rate your comfort level with the READ 180 program model? 
Observations 


Each of the nine intervention teachers was observed up to four times during the 2010-11 
academic year. Observations focused on key characteristics of the READ 180 classroom model 
and teacher instructional practices. As previously described, classroom instruction was rated as 
high, medium, or low implementing on the following aspects of the intervention model. 


Impacts on Teacher Characteristics 


In interviews, reading intervention teachers reported an overall increase in their familiarity with 
and knowledge of the READ 180 program model, as well as increased comfort in using the 
program model. Teachers also indicated knowledge of instructional strategies related to READ 
180 and felt comfortable implementing these strategies. 


In spring interviews, when asked to rate their familiarity with the READ 180 program model, all 
eight reading intervention teachers reported high familiarity, whereas in the fall interviews only 
five teachers rated their familiarity as high and three indicated moderate familiarity. In addition, 
seven of the eight teachers reported a high comfort level with the READ 180 program model in 
spring interviews and one indicated moderate. In comparison, in fall interviews, five reading 
intervention teachers reported a high comfort level and three a moderate comfort level. 


Teachers also reported making changes to the rotation block once they became more familiar and 
comfortable with the program model. For example, in spring interviews, six teachers reported 
changes in whole-group instruction from the beginning of the year. These changes included the 
use of new resources and strategies, such as incorporating the Red Routines, and some teachers 
reported incorporating their own activities, such as writing prompts. Three teachers specifically 
mentioned that as the academic year progressed that they became calmer and more comfortable 
with the READ 180 program model. 


Through interviews, teachers also reported increases in knowledge related to instructional 
strategies, as well as comfort in implementing these strategies as the year progressed. At various 
points in each interview, all teachers reported using student engagement activities such as 
reading quizzes to keep students engaged in the independent reading rotation or reflection circles 
to help students make connections from their readings in the whole-group wrap-up. In spring 
interviews, all teachers reported frequently accessing and reviewing READ 180 reports as well 
as using assessment data to inform instruction. Specifically, teachers reported using assessment 
data, including SAM data and SRI lexile scores to help inform their student groupings for small- 
group instruction. For example, in spring interviews, one teacher explained 


I plan like if I look on what they’re doing on the computer I notice most of these 
students are struggling in main idea and detail, I group those kids together and 
that’s what our focus will be. I may group them together based on their SRI 
scores, these are the basic minimal proficient, I’ll put those students together. 
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Impacts on Teacher Instructional Practices 


Not only were there impacts on the teacher’s attitudes toward READ 180, observation data and 
interviews show some changes in teacher’s instructional practices throughout the year. For 
instance, three of the classrooms improved from a low/medium READ 180 observation score to 
high across the year. Seven of eight classrooms scored in the high range during the last 
observation. 


In interviews, reading intervention teachers reported several positive changes to their 
instructional practices. By the spring, nearly all (seven of eight) teachers reported that they were 
able to incorporate all components of the READ 180 model, as compared with only five of eight 
teachers in the fall. Observation data further revealed that four teachers showed improvement in 
their transitions, which may have led to teachers’ ability to incorporate all components. 


Teachers also reported frequently accessing and reviewing READ 180 reports to plan 
instructional strategies and group their students, and an increase in their ability and 
understanding of interpreting results. One teacher reflected on her improvement in use of data: 


Yes, because I now know what’s important on the data.... [T]ypically, you just 
look for who’s struggling and who’s not, but now I know how to read in between 
the lines too.... [SAM and SRI data] help me with instructional strategies because 
then I can go in and instead of just saying, I need to re-teach, I know exactly what 
to reteach or what I may have missed altogether. What I don’t need to bother with 
anymore because they’ve got it. How to group the students accordingly. 


Two teachers mentioned that the use of a reading quiz helped in the effectiveness of independent 
reading time. As one teacher explained, “Since I started incorporating those printout tests, it’s 
gotten them really interested. In fact, it’s gotten to the point where I’ve got to slow them down.” 


Nearly all teachers noted some changes in their whole-group instruction pertaining to their 
flexibility with activities, and using appropriate activities for different groups of students. As one 
teacher stated, 


Yes. I have one class who really doesn’t like to get up and move a lot, so I allow 
them to do more seatwork. But then I have two blocks that like to do the different 
activities, so we do more up and around. I think having small class sizes, I can 
adjust for their needs and how they learn best. At the beginning of the year I 
didn’t know that, so I had to figure that out during the course of the year. 


Other changes noted throughout the year included better differentiation of students, granting 
more freedom to students as the year progressed, and creating groups consisting of higher 
level students in the reading rotation. All teachers reported using instructional software in the 
spring, as compared with the fall, when three reading intervention teachers reported frequently 
eliminated the instructional software rotation as a result of malfunctions or not having enough 
equipment. 
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Six of eight teachers reported changes specific to the whole-group wrap-up; these included 
having a drawing at the end of the day, varying approaches to the whole group (writing one day, 
having students summarize verbally another day), and tossing around a ball with summarizing 
questions. 


Perceptions of Student Behavior Changes in the Classroom 


Interviews and observations teachers provide some details about perceived changes in student 
behavior in the classroom, though there was some variation in perceptions of READ 180’s 
impact on student behavior. In interviews, the majority of teachers mentioned that their students 
had more confidence as a result of the READ 180 program and showed a greater motivation to 
read. Further, the majority of teachers, in spring interviews, reported that the READ 180 program 
has been helpful for keeping students on task. As one teacher stated, “It has an effect on student 
behavior. They feel better about themselves and they behave more consistently. It has all to do 
with self-confidence. Now they are learning that they can read and I think it makes them feel 
better about themselves.” Principals also agreed that READ 180 positively affected student 
behavior, as one stated, 


I think it has a tremendous impact in a positive manner because it reduces the 
number of incidents that you have because students are on task, time on task. If 
students are busy, you don’t have time to be messing around, getting into extra 
negative curricular activities, if you will. So the students are engaged and that’s a 
positive. 


Still, three teachers felt the READ 180 program did not make a difference in their students’ 
behavior and explained that the kids would show the same behavior no matter what. As one 
explained, 


If you were to go to other classes and observe them, they’re doing much the same 
thing. They’re running around, they’re loud. They’re inappropriate. It’s just the 
nature I guess of a city school and struggles they have with getting kids in their 
school that understand their role. They just don’t understand their role. It’s like 
they’re being punished. Especially the eighth graders, they come in here very 
resentful that their friends are playing basketball in gym and they’re here. 


In general, observation items on behavior were consistent across the year at a medium to high 
level, with one exception. For smooth transitions between all models, four classrooms showed 
improvement from low or medium to high ratings by the last observation, suggesting better 
behavior of students during transitions. Still observation data on effectiveness in addressing 
behavioral issues, clear instructions and outline of expectations, mutual respect among teachers, 
and disrespectful behaviors between teacher and students remained consistent at a medium to 
high level. 


Impacts on Students at the End of One Year 


The statistical analysis of student outcomes focused on the impact of READ 180 on students’ 
reading achievement and engagement. The original study focused on measuring the impact of a 
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two-year reading intervention on students in Grades 6-10. Because the Striving Readers grant 
was discontinued, this study will focus on the impact of a one-year reading intervention on 
students in Grades 6-9. After the main impact modeling, the impact of READ 180 will be broken 
down by both school level—middle school (Grades 6-8) versus high school (Grades 9-10)—and 
by student subgroup. The two main subgroup of interest for this study are special education 
students. These analyses can be found under the heading Additional Analyses. 


Measures of Student Achievement Outcomes 


This section will summarize the student outcomes measures collected for this study—the 
Northwest Evaluation Association Measures of Academic Progress (MAP) and the student 
survey. 


Student Reading Proficiency 


The Measures of Academic Progress (MAP) assessment served as the primary outcome measure 
of student reading proficiency for the analytic analysis. Developed by the Northwest Evaluation 
Association, the MAP is a computer-adaptive assessment aligned to state standards that provides 
the school with immediate feedback on student progress.'” Student scores across grade levels are 
on a developmental metric, allowing schools to follow student progress from one grade to the 
next. In the 2010—11 school year, Milwaukee Public Schools began to identify the MAP as the 
primary assessment measure of student progress, replacing the ThinkLink identified as the 
primary outcome of interest in the original proposal. The MAP is administered to all MPS 
students in mathematics and reading three times a year, October, February, and June. For the 
purposes of this study, the reading portion on the June MAP assessment will be used as the 
primary outcome measure for student reading proficiency. 


Reading Engagement and Self-Efficacy 


American Institutes for Research developed a student survey to determine whether participation 
in the READ 180 reading intervention had an impact on student engagement and self-efficacy 
related to reading. The survey asked students to respond to items related to self-efficacy and the 
constructs of behavioral engagement, emotional engagement, and cognitive engagement with 
reading. 


Impacts on Student Reading Proficiency 


Table 24 shows the estimated intent-to-treat (ITT) impact of one year of a READ 180 
intervention on student reading proficiency. All models include the randomization stratification 
variables (i.e., all combinations of randomization period, school, and grade). The first model (1) 
includes only treatment status as a predictor, the second model adds a pretest and indicator 
variables for missing pretests, and the third model includes student covariates. Using a backward 
selection criterion with a critical value p < 0.20 to select covariates, only special education status 
and other race/ethnicity (i.e., Native American and Asian American) remained in the model. As 


10 http://www.nwea.org/products-services/computer-based-adaptive-assessments/map 
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shown, treatment students scored approximately 1.8 points higher than control students when 
controlling for pretest scores and student-level covariates. This represents an effect size of 

d = 0.138 (calculated as the treatment effect divided by the standard deviation of the control 
group scores). 


Table 24. ITT Estimates on Reading Proficiency 


(1) (2) (3) 
Intercept pranctl ee ae 
P (2.77) (7.99) (8.16) 
Treatment ees me nee 
(1.01) (0.91) (0.88) 
0.10** 0.07** 
Pretest: WKCE (0.01) (0.01) 
0.15 0.11** 
Pretest: BM (0.02 (0.02) 
; 2.94 
Other (1.70) 
—7.78** 
SPED (1.14) 
N 619 619 619 
R 0.20 0.37 0.43 


Note: Parentheses show robust standard errors. All school-by-grade-by-randomization 
blocks and missing variable indicators were included in the models. 

“ “Other” student ethnic groups were Native American and Asian American students. 
** np < 0.01 

*p < 0.05 


Treatment on Treated (TOT) Estimate 


Table 25 shows the estimated impact of one year of READ 180 intervention on student reading 
proficiency. The TOT estimate is a locale average treatment effect that identifies the impact of 
READ 180 for students who enrolled in READ 180 on the basis of their treatment status. As 
shown, treatment students who attended READ 180 scored approximately 2.4 points higher than 
control students who did not attend READ 180 when controlling for pretest scores and student- 
level covariates selected for the ITT analysis. This represents an effect size of d = 0.184. These 
results suggest that the effect of treatment is somewhat diluted in the ITT analysis in that many 
students assigned to enroll in READ 180 did not end up receiving treatment. 
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Table 25. Treatment on the Treated Estimates on Reading Proficiency 


(1) (2) (3) 
ae 192.39** 113.80** 137.87** 
P (2.78) (7.66) (7.92) 
1.80 2.47* 2.38* 
In READ 180 (1.34) (1.20) (1.15) 
0.10** 0.07** 
Pretest: WKCE (0.01) (0.01) 
0.15** 0.11** 
Pretest: BM (0.02) (0.02) 
; 2.79 
Other (1.69) 
—7.61** 
SPED (1.11) 
N 617 617 617 
R 0.21 0.38 0.43 


Note: Parentheses show robust standard errors. All school-by-grade-by-randomization 
blocks and missing variable indicators were included in the models. 

“ “Other” student ethnic groups were Native American and Asian American students. 
** np < 0.01 

*p < 0.05 


Impacts on Student Self-Efficacy and Motivation 


In addition to achievement outcomes, the study sought to determine if participation in the READ 
180 reading intervention had an impact on student engagement and self-efficacy related to 
reading. 


Overall, results of the student survey were non-conclusive, with students from the control and 
treatment groups responding similarly to the majority of survey items, though treatment students 
did indicate slightly higher self-efficacy related to reading as well as higher levels of behavioral 
engagement. For example, treatment students had more positive perceptions of their reading 
ability than students in the control group and treatment students seemed to indicate an increased 
desire to read. Treatment students’ responses to the survey also indicate that they are better able 
to apply strategies to understand what they are reading than those students in the control group. 
The complete results of the Student Engagement Survey can be found in Appendix E. 


Self-Efficacy 


Students responding to the Student Engagement survey were asked to rate their level of 
agreement with six statements related to self-efficacy and reading, shown in Table 26. Their 
responses to the six survey items addressing this construct were then combined into a single 
scale score. These scale scores where then translated to an overall level of agreement with the 
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items in the scale.'' Students were categorized according to their typical response to these six 
items. 


Table 26. Self-Efficacy Survey Items 


Self-efficacy Survey Items 


1. can read difficult material. 


10. Ihave the reading skills I need to complete my school work 


15. My reading ability is above average. 


20. Iam becoming a better reader. 


24. I will be a good reader when I graduate from high school. 


28. Iam a good reader. 


As summarized in Table 27, students in the treatment group were more likely to either agree with 
items related to self-efficacy and reading than students in the control group. 


Table 27. Overall Ratings Across Self-Efficacy Survey Items 


Disagree/ Snmerchet 
Group Somewhat Agree N 
, Agree 
Disagree 
Control 11.0% 53.4% 35.6% 118 
Treatment 8.4% 46.6% 45.0% 131 


Students’ responses to each of the individual items within the construct of self-efficacy also were 
analyzed (see Appendix E). Pearson’s chi-square was used to assess whether differences between 
the responses of students in the control group and treatment group to each item were statistically 

significant. A summary of responses that were significantly different at the 0.05 level follows. 


As shown in Table 28, although both treatment and control students were confident that they will 
be good readers when they graduate from high school, significantly more treatment students (95 
percent) indicated that they agreed or somewhat agreed with this statement than control students 
(87 percent). 


Table 28. I will be a good reader when I graduate from high school, 


N=241 
eset Somewhat Agree 
N Somewhat 8 
; or Agree 
Disagree 
Control 116 12.9% 87.1% 
Treatment 125 4.8% 95.2% 


"' The categories of disagree and somewhat disagree were collapsed on the basis of the psychometric analysis of the 
survey. 
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Differences between the control and treatment groups in students’ agreement with the statement 
“Tam a good reader” also were found to be statistically significant. As shown in Table 29, 
treatment students more frequently agreed or somewhat agreed that they were good readers. 


Table 29. I am a good reader, N = 243 


Disagree or 
N Cae ae Somewhat Agree 
; or Agree 
Disagree 
Control 116 18.1% 81.9% 
Treatment 127 8.7% 91.3% 


Motivation and Engagement 
Through the survey, students also were asked to rate their agreement with items that related to 
three subcategories of motivation and engagement in reading: behavioral engagement, emotional 


engagement, and cognitive engagement. 


Table 30. Student Engagement and Motivation Survey Items 


Behavioral Engagement Survey Items 


2. Tusually finish books that I start. 


5. [sometimes seek out books or articles to read on my own time. 


11. Iusually complete reading assignments for school. 


16. I tend to read during my free time. 


21. I usually try to read carefully to understand what I am reading. 


25. If lam reading an interesting book or article I sometimes lose track of time. 


Emotional Engagement Survey Items 


3. I feel comfortable reading aloud in front of other classmates. 


6. If my teacher told me I was a good reader, I would feel proud. 


8. I enjoy reading in my free time. 


12. If my parents told me I was a good reader, I would feel proud. 


13. Reading information online is fun. 


17. My friends or family are interested in what I am reading. 


18. Reading fiction is fun (for example: novels and fantasy). 


22. I enjoy reading nonfiction (for example: biographies and history). 


26. Reading for school assignments can sometimes be interesting. 
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Cognitive Engagement Survey Items 


4. sometimes summarize what I’ve read to help me better understand important 
information in the reading material. 


7. If what I am reading becomes difficult, sometimes I try to pay closer attention so I 
can better understand what it means. 


9. Reading is important for getting good grades. 


14. Being a good reader will be important after I graduate from high school. 


19. Sometimes I discuss what I read with others to check my understanding. 


23. I know how to try to better understand what I am reading. 


27. 1 sometimes look over the reading material to see what it is about before reading. 


As with responses to items for the construct of self-efficacy, students’ responses to the survey 
items addressing each construct presented in Table 30 were then combined into a single scale 
score for each and students were categorized according to their typical response to the items 
within the construct. 


Behavioral Engagement 


As summarized in Table 31, students in the treatment group were more likely to either agree with 
items related to their behavioral engagement in reading than students in the control group. 


Table 31. Overall Ratings Across Behavioral 
Engagement Survey Items 


Disseree Somewhat 
Group Somewhat Agree N 
: Agree 
Disagree 
Control 30.5% 38.1% 31.4% | 118 
Treatment 26.0% 30.5% 43.5% | 131 


Overall, when students’ responses to each of the individual items within the construct of 
behavioral engagement were analyzed, responses of the students from the treatment group were 
similar to those of students in the control group (see Appendix E). Both treatment and control 
students agreed that they tend to read carefully to understand what they are reading—90 percent 
of control students and 84 percent of treatment students reported that they agreed or somewhat 
agreed with this item—and the majority of students within both groups agreed that they usually 
complete reading assignments for school. 


As shown in Table 32, a higher percentage of students from the treatment group indicated that 
they either agreed or somewhat agreed that they sometimes seek out books or articles to read on 
their own time. The differences in responses between the two groups were found to be significant 
and seem to indicate a greater desire to read among treatment students than among control 
students. 
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Table 32. I sometimes seek out books or articles to read 
on my own time, N = 247 


Disagree or ean 
Af Somewhat Agree or Agree 
Disagree g 8 
Control 116 37.9% 62.1% 
Treatment 131 22.1% 77.9% 


Emotional Engagement 


As summarized in Table 33, students in the treatment group were more likely to either agree or 
somewhat agree with items related to their emotional engagement in reading than students in the 
control group. 


Table 33. Overall Ratings Across Emotional 
Engagement Survey Items 


Jt seas Somewhat 
Group Somewhat A Agree N 
: gree 
Disagree 
Control 31.7% 28.8% 39.4% 104 
Treatment 24.2% 32.3% 43.5% 124 


When individual items within the construct of emotional engagement were analyzed, no 
significant differences were found. It is interesting to note that both treatment and control 
students agreed that if their teachers told them they were good readers, they would feel proud. 
Similarly, the majority of both groups of students indicated that they either agreed or somewhat 
agreed that they would feel proud if their parents told them they were good readers. 


Cognitive Engagement 
As summarized in Table 34, students in the treatment group were more likely to either agree or 
somewhat agree with items related to their cognitive engagement in reading than students in the 


control group. 


Table 34. Overall Ratings Across Cognitive 
Engagement Survey Items 


Denes Somewhat 
Group Somewhat A Agree N 
‘ gree 
Disagree 
Control 11.0% 47.5% 41.5% | 118 
Treatment 11.5% 33.6% 55.0% | 131 
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Overall, students responded most positively to items on the survey related to cognitive 
engagement— two thirds or more of treatment students and at least 60 percent of students from 
the control group either agreed or somewhat agreed with each of the individual items within the 
construct. 


Teachers’ Perceptions of Student Development in Reading Skills 


In interviews, teachers and principals asserted that READ 180 had an impact on students’ 
reading skills, including their self-efficacy, motivation, and level of achievement. All teachers 
perceived that they observed increases in student confidence and greater motivation to read in 
their READ 180 student. In the words of one teacher, 


It has an effect on student behavior. They feel better about themselves and they 
behave more consistently. It has all to do with self-confidence. Now they are 
learning that they can read and I think it makes them feel better about themselves. 


Interviews with principals of the study schools also indicated increased motivation to read, with 
one principal explaining, “I think that some students did develop more confidence and therefore 
became a little more motivated because they saw the progress.” 


Each of the five study school principals also reported that READ 180 positively affected student 
achievement; as one stated, 


We’ ve seen a number of students who eventually worked their way out of the 
program and transitioned into a regular reading program. We’ve seen some 
significant gains not only from the READ 180 data, but it actually aligned to the 
data that we were using for our benchmark assessments at the district and school 
level. So we were seeing those same groups of students making gains on those 
assessments. 


Discussion/Conclusions 


Overall, there is some quantitative and qualitative evidence that a one-year READ 180 
intervention improves students reading proficiency as well as self-efficacy and motivation. 
Results from the ITT analysis found that students assigned to the treatment condition scored on 
average two points higher than students assigned to the control condition when controlling for 
pretest scores and student-level covariates. In addition, some statistical differences were found 
on student surveys: a greater percentage of treatment students than control students reported 
having greater self-efficacy and behavior, emotional, and cognitive motivation to read. Last, in 
interviews, all teachers and principals confirmed assessment and survey data that READ 180 had 
an impact on students’ reading skills and had impact of the program on students’ self-efficacy, 
motivation, and achievement. 


American Institutes for Research Striving Readers Impact Study—59 


Additional Analyses 


This section includes experimental analyses on special education students, ELL learners, and 
middle school versus high school students; the treatment on the treated analysis; and dosage 
study. 


Experimental Analyses 
Special Education Status 


An experimental analysis was run on special education students assigned to treatment using an 
interaction term between special education and treatment status. As shown in Table 35, no 
Statistical differences were found in reading proficiency between special education students 
assigned to READ 180 and special education students not assigned to READ 180 when 
controlling for pretest scores and other student-level covariates determined by the backward 
selection criteria used for the ITT analysis. 


Table 35. ITT Estimates for Special 
Education Students in READ 180 


(3) 
Intercept oe cia 
P (8.23) 
2.30* 
Treatment (1.04) 
—1.45 
x 
Treatment*SPED (1.84) 
0.07** 
Pretest: WKCE (0.01) 
0.11** 
Pretest: BM (0.02) 
a 3.06 
Other (1.75) 
—7.02** 
pee (1.54) 
N 619 
R 0.43 


Note: Parentheses show robust standard errors. All 
school-by-grade-by-randomization blocks and 
missing variable indicators were included in the 
models. 


“ “Other” student ethnic groups include Native 
American and Asian American students. 


** pn <0.01 
*p < 0.05 
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English Language Learner Status 


An experimental analysis was run on ELLs assigned to a treatment condition relative to ELLs 
assigned to the control condition using an interaction term between ELL and treatment status. As 
shown in Table 36, no statistical differences were found in reading proficiency between ELL 
students assigned to READ 180 and ELL students not assigned to READ 180 when controlling 
for pretest scores and other student-level covariates determined by the backward selection 
criteria used for the ITT analysis. 


Table 36. ITT Estimates for English 
Language Learner Students in READ 180 


(3) 
137.42** 
Intercept (8.22) 
Treatment cal 
(0.96) 
—2.61 
oK 
Treatment* ELL (2.39) 
0.07** 
Pretest: WKCE (0.01) 
: 0.11** 
Pretest: BM (0.02) 
a 2.90 
Other (1.73) 
—7.81** 
eee (1.13) 
1.35 
ee (2.08) 
N 619 
R 0.43 


Note: Parentheses show robust standard errors. All 
school-by-grade-by-randomization blocks and 
missing variable indicators were included in the 
models. 


“ “Other” student ethnic groups include Native 
American and Asian American students. 


** pn <0.01 
*p < 0.05 
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Middle School Versus High School Students 


Table 37 shows the ITT estimates for middle school (Grades 6-8) and high school (Grade 9) 
students who participated in the study. As shown, no significant differences were found between 
students in the treatment and those in the control condition in either the middle school or the high 
school sample. In addition, although not significant, both middle school and high school students 
in the treatment condition scored approximately two points higher than students in the control 
condition. The nonsignificance of these results is likely due to a lack of statistical power for this 
test because only one year of data was collected (instead of the three planned for the study). 


Table 37. ITT Estimates for Middle School Versus High 
School Students, Model 3 


Middle School High School 
eee 137.82** 1247.70** 
P (9.40) (19.231) 
Treatment mate cof 
(1.01) (1.83) 
0.06** 0.07* 
Pretest: WKCE (0.02) (0.03) 
0.12** 0.16** 
Pretest: BM (0.03) (0.05) 
7 2.68 4.99 
Ores (2.03) (3.39) 
—7.52** —7,.73** 
OnE (1.3) (2.15) 
N 480 139 
R 0.40 0.53 


Note: Parentheses show robust standard errors. All 
school-by-grade-by-randomization blocks and 
missing variable indicators were included in the 


models. 


“ “Other” student ethnic groups include Native 


American and Asian American students. 


** pn <0.01 
*p < 0.05 
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Dosage Study 


The dosage study examined whether attendance and fidelity of SAM usage affected the reading 
proficiency of treatment students. As shown in Table 38, students with medium and high 
attendance in READ 180 did not score significantly higher than students with low attendance 
(although the effect for high attendance is close to being significantly different from low use). 
Nevertheless, it appears that scores for students who attended more were higher on average than 
those for students who attended less (conditional on prior achievement). Likewise, as shown in 
Table 39, students with medium and high SAM usage did not score significantly higher than 
students with low SAM usage. 


Table 38. Treatment Students Attendance 
and Reading Proficiency 


(3) 
Intercept ea. 
Medium Attendance Aas 
High Attendance nee 
Pretest: WKCE coe 
Pretest: BM oa. 
Other* ee 
SPED (a). 
N 326 
Re 0.56 


Note: Parentheses show robust standard errors. All 
school-by-grade-by-randomization blocks and 
missing variable indicators were included in the 
models. 


“ “Other” student ethnic groups include Native 
American and Asian American students. 


** pn < 0.01 
*p < 0.05 
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Table 39. Treatment Students Fidelity 
With SAM Usage and Reading 


Proficiency 
(3) 
137.63** 

Intercept (12.41) 
Medium SAM Use ree 
High SAM Use a8) 
2k 
Pretest: WKCE (0G) 
2k 
Pretest: BM (605 
Other* ou 
-8.31** 
prep (1.44) 

N 326 

R? 0.55 


Note: Parentheses show blocks and missing variable 
indicators were included in the models. 


“ “Other” student ethnic groups include Native 
American and Asian American students. 


** pn <0.01 
*p < 0.05 
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Conclusion 


This report provides the details of Milwaukee Public Schools’ Striving Readers program, the 
fidelity of its implementation, and results of that implementation. As part of their Striving 
Readers grant, Milwaukee provided supplemental instruction to treatment students using READ 
180, a comprehensive reading intervention program developed by Scholastic. This program is 
designed to address individual student needs through adaptive instructional software, high- 
interest literature, and direct instruction in reading and writing skills. 


This study was designed with two main research objectives: (1) examine the fidelity of 
implementation of the classroom and professional development model; and (2) identify any 
impact of the program on student outcomes, including student reading achievement. To 
accomplish this research, AIR implemented a student-randomized trial of READ 180 in five 
Milwaukee schools. Treatment students were to receive the regular ELA course plus 
supplemental reading instruction in a 90-minute READ 180 course. Control students were to 
receive regular ELA instruction plus an elective or study hall period. 


AIR’s study found that READ 180 teachers in the five schools implemented the classroom model 
with high fidelity. Student attendance was an issue, however. Although good instruction was 
happening in the classroom, many treatment students were not in class to receive that instruction. 
Teachers also did not receive all the professional development and mentoring whose 
implementation was intended. 


Although implementation fidelity was not perfect, the study still found a statistically significant 
improvement for the treatment students in reading achievement on the NWEA MAP assessment. 
Furthermore, there seem to be slight indications that students who received READ 180 
instruction may self-report higher engagement with reading activities, and higher motivation to 
read. 
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WDPI/MPS Striving Readers Interview Rubric 


General Information 


Appendix A 


High 


Rating Level 


Medium 


Low 


Interview 
Questions 


Length of time at 
school, school 
experience 


Note length of time at school 


Note length of time as teacher/administrator 


Fall Tchr: 1,1a, 
2 
Spring Tchr:1, 
2 
Principal: 1,1a 


Previous 
experience with 
other struggling 
readers programs 


Note level of experience with previous programs related to 


struggling readers. 


Note how that previous experience relates to the Read 180 


program. 


Fall Tchr: 3, 3a 
Principal: 4, 4a 


Familiarity with 
Read 180 
program model 


Teachers report 
that they have a 
high extent of 
familiarity with the 
Read 180 program 
model. 


Teachers report 
that they have a 
moderate extent of 
familiarity with the 
Read 180 program 
model. 


Teachers report 
that they have a low 
extent or no 
familiarity with the 
Read 180 program 
model. 


Fall Tchr: 2a 
Spring Tchr: 2a 


Comfort with 
Read 180 
program model 


Teachers report 
that they have a 
high extent of 

comfort with the 


Teachers report 
that they have a 
moderate extent of 
comfort with the 


Teachers report 
that they have a low 
extent or no 
comfort with the 


Fall Tchr: 2b 
Spring Tchr: 2b 


School vision 
related to Read 
180 program 


Read 180 program | Read 180 program | Read 180 program 
model. model. model. 
The Read 180 The Read 180 tne Reg tae 


program is a 
primary focus of the 
school vision 


program is a minor 
focus of the school 
vision 


program is not 
integrated into or 
runs counter to the 
school vision 


Principal: 2 


Other current 
literacy programs 
for struggling 
readers (Could be 
N/A) 


The Read 180 
program is fully 
integrated with 
other literacy 
programs and/or 
successfully serves 
complimentary 
purposes 


The Read 180 
program is 
somewhat 
integrated with 
other literacy 
programs and/or 
moderately serves 
complimentary 
purposes 


The Read 180 
program runs 
counter to other 
literacy programs 
at the school and/or 
has conflicting 
purposes 


Principal: 3 
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Program Setup 


Class size and 
distribution 


Rating Level 


High 


The class size is 
appropriate for all 
students and grade 
levels 


Medium 


The class size is 
appropriate for 
some students and 
grade levels but not 
others 


Low 


The class size is 
not appropriate for 
the number of 
students and 
different grade 
levels 


Interview 
Questions 


Fall Tchr: 4, 4a, 
4b, 5, 6 

Spring Tchr: 3, 
3a, 3b, 4,5 
Principal: 5, 5a 


Student schedules 
and classroom 
space 


There were few if 
any challenges 
with student 
schedules and 
classroom space, 
and/or the school 
was very 
successful in 
addressing these 
issues 


There were some 
challenges with 
student schedules 
and classroom 
space, and the 
school was 
moderately 
successful in 
addressing these 
issues 


There were many 
challenges with 
student schedules 
and classroom 
space, and/or the 
school did not 
address these 
issues 


Principal: 6, 6a 


Implement full 
Read 180 three- 
part structure 


Teachers are 
consistently able to 
incorporate all 3 
sections of the full 
Read 180 
structure. Note 
successful 
strategies. 


Teachers are 
inconsistently able 
to incorporate all 3 
sections of the full 
Read 180 structure. 
Note successful 
strategies and 
challenges faced. 


Teachers are not 
able to incorporate 
all 3 sections of 
the full Read 180 
structure. Note 
challenges faced. 


Fall Tchr: 7, 7a, 
7b 

Spring Tchr: 6, 
6a, 6b 
Principal: 7, 7a, 
7b 


Smooth transitions 
between all 
models (e.g. 
Timer, two-minute 
warning) 


Teachers report 
that there are 
smooth transitions 
during all activity 
rotations. 


The teacher 
always uses a 
timer, two minute 
warning, and/or 
other method to 
indicate to 
students of a 
rotation change. 
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Teachers report 
that there are some 
smooth transitions 
during the activity 
rotations and/or 
transitions vary 
from class to class. 


The teacher 
sometimes uses a 
timer, two minute 
warning, and/or 
other method to 
indicate to students 
of a rotation 
change. 


Teachers report 
that transitions are 
difficult during the 
activity rotations. 
The teacher never 
uses a timer, two 
minute warning, 
and/or other 
method to indicate 
to students of a 
rotation change. 
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Fall Tchr: 8 
Spring Tchr: 7 


Rating Level Interview 
High Medium Low Questions 
Structure students pene Ruse paneer Teachers report 
on task ; having difficulty 
the Read 180 successful using the with the Read 180 | Fall Tchr: 9 


(i.e. Students are 
involved and 
engaged in the 
task at hand as 


structure to keep 
the majority of 
students on task 
for the majority of 


Read 180 structure 
to keep the majority 
of students on task 
for the majority of 


structure to keep 
students on task 
for the majority of 


Spring Tchr: 8 
Principal: 8 


expected) Hidtime he tine the time. 
Materials/Technology 
Rating Level Interview 
High Medium Low Questions 
: All materials are Materials are 
at aaa are available, but there | missing, there was 
Materials available and were sen dei ‘ ae, Fall Tchr: 10, 16 
Available / Need | available on time. eae : if Spring Tchr: 14 
sadicenal Phereiedeleak materials. There is | materials, and/or Sean aay 
; asomewhat clear | there is notaclear | *™NCIpal. y, 2, 
materials process to acquire . ; 13 
; process to acquire | process to acquire 
more materials. : : 
more materials. more materials. 
The majority of Some students have Fall Tchr: 11 
: d " he : Few students or : 
Quantity of students have their | their own rBook, enone ee Spring Tchr: 9, 
rBooks own rBook while other 15 
own rBook 
students do not Principal: 10 
READ 180 
There area paperback books at 
sufficient number of | various levels are READ 180 
READ 180 available and paperback books at | Fall Tchr: 12, 13 
Quantity of ; ; : j 
paperback books at | accessible to various levels are Spring Tchr: 10, 
READ 180 ; : 
various levels students; however, | not available and 11,15 
Paperbacks ; : ees 
available and the number of accessible to Principal: 10 
accessible to books at one or students. 
students. more levels is 
insufficient 
There are a Audiobooks are THoreare ne 
sufficient number of | available and Fall Tchr: 12 
Quantity of ae Audiobooks ; : 
Audiobooks accessible; : Spring Tchr: 10, 
READ 180 : available and 
: available and however, the . 15 
Audiobooks : : accessible to Dad 
accessible to number is Principal: 10 
: on students. . 
students. insufficient 
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Functional 
Computers and 
Equipment 


Rating Level Interview 
High Medium Low Questions 
Very few (less than 
Pane Some of the half) of the 
se city eae! computers and computers are 
ah other technology functional. 


other technology 
are functional with 
headsets and 
headphones. 
Teacher has clear 
alternative plans if 
technology fails. 


are functional. 
Some headsets may 
be broken or 
missing. Teacher 
has some 
alternative plans if 
technology fails. 


Accessories are 
missing. Students 
have difficulty 


using the computer. 


Teacher does not 
have alternative 
plans if technology 
fails. 


Fall Tchr: 14, 
14a 

Spring Tchr: 12, 
12a 

Principal: 10, 11 


Use of SMART 
Board 


Teacher 
successfully 
integrates the 
SMART Board with 
the Read 180 
materials. Teacher 
reports feeling 
comfortable and 
knowledgeable 
about the use of the 
SMART Board. 


Teacher sometimes 
integrates the 
SMART Board with 
the Read 180 
materials. Teacher 
reports feeling 
somewhat 
comfortable and 
knowledgeable 
about the use of the 
SMART Board. 


Teacher does not 
integrate the 
SMART Board with 
the Read 180 
materials. Teacher 
reports feeling 
uncomfortable and 
has little 
knowledge about 
the use of the 
SMART Board. 


Whole-Group Instruction 


Fall Tchr: 15 
Spring Tchr: 13, 
15 

Principal: 10, 12 


Teacher approach 
to whole group 
(change from fall 
to spring) 


Rating Level 
High Medium Low 
Teacher reports a Teacher reports a Teacher does not 
clearly defined somewhat defined have a clear 


approach that 
follows the Read 
180 program 
model. 


approach that 
partially follows the 
Read 180 program 
model. 


approach that 
follows the Read 
180 program 
model. 


Interview 
Questions 


Fall Tchr: 17 
Spring Tchr: 16, 
19 


Teacher use of 
materials 


Teacher reports 
frequent use of a 
variety of Read 180 
materials for 
whole-group 
instruction. 


Teacher reports 
some use of Read 
180 materials for 
whole-group 
instruction. Lacks 
variety of 
materials. 


Teacher reports 
not using Read 
180 materials for 
whole-group 
instruction. 


Fall Tchr: 18 
Spring Tchr: 17 
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Teachers use of 


Rating Level Interview 
High Medium Low Questions 
Teacher Teacher tries to 
incorporates incorporate 


instructional 
strategies to 
effectively engage 
the majority of 
students. 


Teacher frequently 
uses Red Routines. 


instructional 
strategies, but it 
does not effectively 
engage the majority 
of students. 


Teacher sometimes 
uses Red Routines. 


Teacher does not 
incorporate 


Fall Tchr: 19, 19a 


instructional O : : instructional 
: Teaching LI Teaching Spring Tchr: 18 

strategies / Use of Vocabulary Vocabulary strategies and 18a 
Red Routines does not use Red 

L) Oral Cloze L) Oral Cloze Routines: 

UL) Think (Write)- | UL Think (Write)- 

Pair-Share Pair-Share 

L) Idea Wave L) Idea Wave 

L) Numbered L) Numbered 

Heads Heads 

U) Writing Process U) Writing Process 

L) Peer Feedback | UL) Peer Feedback 

Rotation Block 
Rating Level Interview 
High Medium Low Questions 

Teacher reportsa_ | Teacher reportsa_ | Teacher does not 
Teacher approach | clearly defined somewhat defined | have a clear Fall Tchr: 20. 20a 
to small group approach that approach that approach that Spring T h o 0 
instruction (change | follows the Read partially follows follows the Read en Sones 
from fall to spring) | 180 program the Read 180 180 program a 

model. program model. model. 


Teacher approach 
to independent 
reading (change 
from fall to spring) 


Teacher reports a 
clearly defined 
approach that 
follows the Read 
180 program 
model. 


Teacher reports a 
somewhat defined 
approach that 
partially follows 
the Read 180 
program model. 


Teacher does not 
have a clear 
approach that 
follows the Read 
180 program 
model. 


Fall Tchr: 20, 
20b 

Spring Tchr: 20, 
20b, 21 


Teacher approach 
to Read 180 
software time 
(change from fall 
to spring) 


Teacher reports a 
clearly defined 
approach that 
follows the Read 
180 program 
model. 


Teacher reports a 
somewhat defined 
approach that 
partially follows 
the Read 180 
program model. 


Teacher does not 
have a clear 
approach that 
follows the Read 
180 program 
model. 


Fall Tchr: 20, 20c 
Spring Tchr: 20, 
20c, 21 
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Student grouping 


Rating Level Interview 
High Medium Low Questions 
TeGener has Teacher has some_ | Teacher has no 
clearly defined aaAnadarti en Aohicd een 
ériteria for efined criteria for | defined criteria for Fall Tchr: 21, 21a 


creating student 
groups and uses 
several data 

sources as input. 


creating student 
groups and uses 
one data source as 
input. 


creating student 
groups and does 
not use any data 
source as input. 


Spring Tchr: 22, 
22a 


Teacher frequently 
provides 
differentiated 
instruction to 


Teacher sometimes 
provides 
differentiated 
instruction to 


Teacher rarely 
provides 
differentiated 
instruction to 


Fall Tchr: 22, 22a 


Differentiated individual students | individual students | individual students ; 
instruction or groups of or groups of or groups of Spring Tchr: 23, 
students based on | students based on | students based on end 
RDI books and RDI books and RDI books and 
other Read 180 other Read 180 other Read180 
materials. materials. materials. 
Teacher reports Teacher reports Teacher reports 
that they that they somewhat | that they do not 
successfully successfully successfully 
Effective manage the manage the manage the Fall Tchr: 23 
independent independent independent independent : 
reading reading time and reading time and reading time and Spring Tchr: 24 
that all students that most students | that few students 
use this time use this time use this time 
effectively. effectively. effectively. 
Teacher reports Teacher reports Teacher reports 
little to no some challenges frequent 
Effective use of challenges with the | with the Read 180 | challenges with the | Fall Tchr: 24, 24a 
Read180 software Read 180 software | software and/or Read 180 software | Spring Tchr: 25, 
and/or all students | most students use and/or few 25a 
use this time this time students use this 
effectively. effectively. time effectively. 
Teacher is 
consistently 
effective in Teacher is rarely 
Effectiveness in addressing Teacher is or not at all Fall Tchr: 25, 25a 


addressing 
behavioral issues 


behavioral issues. 


Or no behavioral 
issues are 
encountered. 


sometimes effective 
at addressing 
behavioral issues. 


effective in 
addressing 
behavioral issues. 


Spring Tchr: 26, 
26a 
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Whole-Group Wrap-Up 


Rating Level Interview 
High Medium Low Questions 
Teacher reportsa_ | Teacher reportsa_ | Teacher does not 
Teacher approach | clearly defined somewhat defined | have a clear 
on Fall Tchr: 26 
to wrap-up activity | approach that approach that approach that Spring Tchr: 27 
(change from fall | follows the Read partially follows follows the Read os 8 et 
to spring) 180 program the Read 180 180 program 
model. program model. model. 
Teacher reports SSE OePeROrE 
that the majority of | Teacher reports h P 
students that some students se OE nS 
Whole Group : : students Fall Tchr: 26a 
summarize and summarize and ; ; 
Reflections 2 : summarize and Spring Tchr: 27a 
share skills from share skills from : 
; dt fa ; ee share skills from 
the day’s activities | the day’s activities er 
the activities 
Teacher reports 
that the majority of Teacher reports 
students complete | Teacher reports 
ey that few or no 
writing that some students Sidon n 
Writing assignments complete writing Aa isleieannah Fall Tchr: 26a 
Reflections covering the day’s_ | assignments P g Spring Tchr: 27a 


reading activities 
(such as Exit Slips 
or Daily Reflection 
Journals) 


covering the day’s 
reading activities 


assignments 
covering the day’s 
reading activities 
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Data and Reports 


Teacher use of 
assessment data 


Rating Level 
High Medium Low 
Teachers Teachers 


frequently access 
student data, 
review this 
information, and 
use it for planning. 


sometimes access 
student data, 
review this 
information, and 
use it for planning. 


Teachers rarely 
access student 
data, review this 
information, and 
use it for planning. 


Interview 
Questions 


Spring Tchr: 29, 
29a 

Principal: 14, 14a, 
14b 


Teacher use of 
Read 180 reports 


Teachers 
frequently access 
Read 180 
generated reports, 
review this 
information, and 
use it for planning. 


Teachers 
sometimes access 
Read 180 
generated reports, 
review this 
information, and 
use it for planning. 


Teachers rarely 
access Read 180 
generated reports, 
review this 
information, and 
use it for planning. 


Spring Tchr: 30, 
30a 

Principal: 14, 14a, 
14b 


Share data, teacher 
meetings about 
data 


Teachers 
frequently share 
student progress 
data with 
administrators, 
coaches, core 
teachers, and 


Teachers 
sometimes share 
student progress 
data with 
administrators, 
coaches, core 
teachers, and 


Teachers rarely 
share student 
progress data with 
administrators, 
coaches, core 
teachers, and 


Spring Tchr: 31, 
31a, 32 
Principal: 15, 16, 
17 


parents. parents. DaLents. 
Instructional Support 
Note: Rating Level 
Any support : 
could be N/A if ' : Interview 
not available to | High Medium Low Questions 


the teacher. 


Principal 
support—Quantity 


The administrator 
provides frequent 
support for the 


Read 180 program. 


The administrator 
provides some 
support for the 
Read 180 program. 


The administrator 
provides little 
support for the 
Read 180 program. 


Fall Tchr: 27, 32 
Spring Tchr: 33, 
38, 39 

Principal: 18, 23, 
24 


Principal 
support—Quality 


The administrator 
has affected Read 
180 instruction to 
a large extent. 


The administrator 
has affected Read 
180 instruction to 
a moderate extent. 


The administrator 
has affected Read 
180 instruction to a 
small extent or not 
at all. 


Fall Tchr: 27, 32 
Spring Tchr: 33, 
38, 39 

Principal: 18, 23, 
24 
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Note: 

Any support 
could be N/A if 
not available to 
the teacher. 


In-school support 
(e.g. Instructional 
coach)—Quantity 


Rating Level 


High 


The in-school 
support person 
provides frequent 
support for the 


Read 180 program. 


Medium 


The in-school 

support person 
provides some 
support for the 


Read 180 program. 


Low 


The in-school 
support person 
provides little 
support for the 
Read 180 program. 


Interview 
Questions 


Fall Tchr: 28, 32 
Spring Tchr: 34, 
38, 39 

Principal: 19, 23, 
24 


In-school support 
(e.g. Instructional 
coach)—Quality 


The in-school 
support person has 
affected Read 180 
instruction to a 
large extent. 


The in-school 
support person has 
affected Read 180 
instruction to a 
moderate extent. 


The in-school 
support person has 
affected Read 180 
instruction to a 
small extent or not 
at all. 


Fall Tchr: 28, 32 
Spring Tchr: 34, 
38, 39 

Principal: 19, 23, 
24 


anaes The district The district The district support | Fall Tchr: 29, 32 
a ai oo support person support person person provides Spring Tchr: 35, 
we dinator)— provides frequent | provides some little support for 38, 39 
Ouentit support for the support for the the Read 180 Principal: 20, 23, 
y Read 180 program. | Read 180 program. | program. 2A 
ite The district The district The district support | ral] Tchr: 29, 32 
Pe Ue nO support person has | support person has | P&'°°” Has affected Spring Tchr: 35 
(e.g. is eee Dee Read 180 Pe Baie 
epordinaton= affected Read 180 | affected Read 180 Scone 38, 39 
Quality instruction to a instruction to a araiheenton vee, | pumcipalt.20,.23; 
large extent. moderate extent. Heel 24 
The 
The The 
Fall Tchr: 30, 32 
Scholastic/Read Scholastic/Read Scholastic/Read Fi sacne Spring Tchr: 36 
180 180 representative | 180 representative bakes ae PPS 
ve Suis) ‘d : provides little 38, 39 
presentative provides frequent | provides some support for the eee 
Quantity support for the support for the R ee 180 prota Principal: 21, 23, 
Read 180 program. | Read 180 program. prog 24 
The 
The The 
Fall Tchr: 30, 32 
Scholastic/Read Scholastic/Read Scholastic/Read es Spring Tchr: 36 
180 180 representative | 180 representative hoes if ted Rend a ao is 
representative— has affected Read | has affected Read 180 EIRIGHGRIE A a8, 
Quality 180 instruction to | 180 instructionto | 1 ad Renee Principal: 21, 23, 
a large extent. a moderate extent. ail 24 
Other staff Other staff Other staff Fall Tchr: 31, 32 
ee members provide members provide members provide Spring Tchr: 37, 
oan frequent support some support for little support for 38, 39 
y for the Read 180 the Read 180 the Read 180 Principal: 22, 23, 
program. program. program. 24 
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Note: Rating Level 
Any support : 
could be N/A if Slee 
not available to | High Medium Low 
the teacher. 
Other staff Other staff ON Sail Fall Tchr: 31, 32 
members have : 
Oth : members have members have affected Read 180 Spring Tchr: 37, 
a vnee affected Read 180 | affected Read 180 | . ; 38, 39 
Quanity instruction to a instruction to a Se ena rites incipal: 
small extent or not | Principal: 22, 23, 
large extent. moderate extent. 24 
at all. 
Professional Development 
Note: Rating Level 
Boy ED coe Interview 
be N/A if not : ; : 
offered ta High Medium Low Questions 
teachers. 
Teachers fully Teachers Teachers did not Fall Tchr: 33, 
Day 1/ Day 2 oie ; es : ss . 339 41. 42 
Training — participated in the participated in participate in the a, 41, 
Quantit initial Read 180 some of the initial initial Read 180 Principal: 25, 
y training. Read 180 training. | training. 25a, 26, 26a 
Teachers took a lot Teachers took some ; 
of information away _ | information away fe ae Fall Tchr: 33 
Day 1/ Day 2 from the initial Read | from the initial conrad bat 
a hs ie from the initial Read | 33a, 41, 42 
Training— 180 training, and/or | Read 180 training, 180 irdining, and/or | "Principal 95 
Quality reported that it was and /or reported a a he ; uae Be eae : 
of good or excellent _| that it was of fair eral ernoreed 
quality. quality. POOn Aa 
Fall Tchr: 34, 
RED online Teachers fully Teachers Teachers did not 34a, 41, 42 
ae ed participated in participated in participate in any Spring Tchr: 
Ouantit offered RED online some offered RED | offered RED online | 40, 40a, 47, 48 
y courses. online courses. courses. Principal: 25, 
25a, 26, 26a 
ae: Sane — Teachers took some | Teachers took little | Fall Tchr: 34, 
Y | information away _ | information away 34a, 41, 42 
; from the online ; 
RED online ies Gatne from the courses, from the courses, Spring Tchr: 
courses—Quality : . and /or reported and/or reported that | 40, 40a, 47, 48 
REDO eae as that it was of fair it was of poor incipal: 
of good or excellent Sohn Principal: 25, 
quality. quality. 25a, 26, 26a 


quality. 
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Note: Rating Level 
epyee oud Interview 
be N/A if not ; : Oueauie 
offered to High Medium Low 
teachers. 
: Fall Tchr: 35, 
Teachers fully Teachers Teachers did not 35a. 41. 42 
Seminars / participated in participated in participate in S es ie 
conferences— offered Read 180 offered Read 180 offered Read 180 a ct 47 : 48 
Quantity seminars or seminars or seminars or ae 
conferences. conferences. conferences. Principal: 25, 
25a, 26, 26a 
Teachers took a lot Teachers took some | Teachers took little | pay) Tchr: 35 
of information away _ | information away information away 35a. 41. 42 f 
Seminars / from the from the from the e i A 
conferences— seminars/conference | seminars/conferenc_ | seminars/conference ae : 
41, 41a, 47, 48 
Quality s, and/or reported es, and /or reported | s, and/or reported oe od 
that it was of good or | that it was of fair that it was of poor _ | Principal: 25, 
excellent quality. quality. quality. 25a, 26, 26a 
rc Fall Tchr: 36, 
Teachers fully SueeEeee Teachers did not 36a, 41, 42 
Teacher seat | participated in aa : : ie 
Ss et plee= participated in some offered participate in any Spring Tchr: 
: offered teacher offered teacher 42, 42a, 47, 48 
euannty roundtables penebed roundtables incipal: 
roundtables. Principal: 25, 
25a, 26, 26a, 27 
ie cet : = ae as Teachers took little | Fall Tchr: 36, 
Sahel aa |e fort away | information away 36a, 41, 42 
Teacher from the teacher from the teacher : 
from the teacher Spring Tchr: 
roundtables— roundtables, and/or roundtables, and 
: roundtables, and/or | 42, 42a, 47, 48 
Quality reported that they /or reported that we 
reported that they Principal: 25 
were of good or they were of fair were ofnooriquall , 
excellent quality. quality. poor quality. | 25a, 26, 26a, 27 
Fall Tchr: 37, 
SAM Data Teachers fully Teachers Teachers did not 37a, 41, 42 
ea ae participated in participated in participate in Spring Tchr: 
Aaa offered SAM data offered SAM data offered SAM data 43, 43a, 47, 48 
y training. training. training. Principal: 25, 
25a, 26, 26a 
Teachers took alot | Teachers took some | Teachers took little | Fall Tchr: 37, 
SAM Data of information away | information away _ | information away 37a, 41, 42 
ae aay from the training, from the training, from the training, Spring Tchr: 
Cai and/or reported that | and /or reported and/or reported that | 43, 43a, 47, 48 
y it was of good or that it was of fair it was of poor Principal: 25, 
excellent quality. quality. quality. 25a, 26, 26a 
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Note: Rating Level 
Any PD could nteeciets 
be N/A if not ' : Questions 
offeredto High Medium Low 
teachers. 
: Fall Tchr: 38, 
Teachers Teachers Teachers did not 38a 41. 42 
Graduate participated in participated in participate in any S aa ie 
courses— graduate courses graduate courses graduate courses a ee A7 48 
Quantity related to Read 180 | related to Read 180 | related to Read 180 pater 
instruction. instruction. instruction. Principal: 25, 
25a, 26, 26a 
wane ater a Teachers took some | Teachers took little | Fall Tchr: 38, 
Y | information away _ | information away 38a, 41, 42 
from the graduate . 
Graduate from the graduate | from the graduate Spring Tchr: 
: courses, and/or 
courses—Quality courses, and /or courses, and/or 44, 44a, 47, 48 
reported that they oie h A, 
ported that they reported that they Principal: 25 
were of good or : f : 2 
were of fair quality. | were of poor quality. | 25a, 26, 26a 


excellent quality. 


Teachers Teachers Teachers did not Fall Tchr: 39, 
ee in other ones in se ee 39a, 41, 42 
Other PD professiona other professiona other professiona Spring Tchr: 
: development development development 
Quantity a ae a 45, 45a, 47, 48 
opportunities related | opportunities opportunities related eae 
to Read 180 related to Read 180 | to Read 180 Principal: 25, 
instruction. instruction. instruction. 25a, 26, 26a 
pegs rOah cio Teachers took some | Teachers took little | aj) Tchr: 39 
Op injorination aay information awa information awa ener 
from this other id y 39a, 41, 42 
: from other from other : 
Other PD— professional rofessional rofessional Spring Tchr: 
Quality development, and/or P p 45, 45a, 47, 48 
reported that it was development, and development, and/or Ree aes 
of dood orencellent /or reported that it | reported that it was cbeguate Hage 
g° was of fair quality. | of poor quality. 25a, 26, 26a 
quality. 
= a me a Fall Tchr: 40 
Additional Note areas where respondents indicate additional training is ; 
a Spring Tchr: 46 
Training needed. 
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Student Outcomes 


Student 
achievement 


Rating Level Interview 
High Medium Low Questions 
Respondent states | Respondent states 
Respondent states 
that there was a that there was a 
EGEINET eWwas a noticeable noticeable 
noticeable Spring Tchr: 49a 


achievement gain 
made for all or 


achievement gain 
made for some of 


achievement gain 
made for few or 


Principal: 29a 


the students, but none of the 
almost all students. 
not other students. | students. 
Respondent states 
Respondent states P Respondent states 
that there was a 
that there was a . that there was a 
: noticeable ; 
noticeable i AP VERIERT AA noticeable 
; improvement in abe) improvement in Spring Tchr: 49b 
Student behavior ‘ behavior/lack of : ae 
behavior/lack of ; : behavior/lack of Principal: 29b 
: : misbehavior by : : 
misbehavior by all misbehavior by few 
some of the 


or almost all 
students. 


students, but not 
other students. 


or none of the 
students. 


Student self- 
efficacy 


Respondent states 
that there was 
Positive attitude 
and confidence in 
reading expressed 
by all or almost all 
students. 


Respondent states 
that there was 
Positive attitude 
and confidence in 
reading expressed 
by some of the 
students, but not 
other students. 


Respondent states 
that there was 
positive attitude 
and confidence in 
reading expressed 
by few or none of 
the students. 


Spring Tchr: 49c 
Principal: 29c 


Other student 


Note any other noticeable student changes 


Spring Tchr: 49d 


changes Principal: 29d 
Overall Information 

Rating Level Interview 

High Medium Low Questions 
Fall Tchr: 43 

Benefits Note benefits of Read 180 Spring Tchr: 50 
Principal: 30 
Fall Tchr: 44 

Challenges Note challenges of Read 180 Spring Tchr: 51 


Principal: 31 


Recent issues 


Note recent school issues that impacted Read 180 


Fall Tchr: 45 
Spring Tchr: 52 
Principal: 32 
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Classroom Fidelity Score Weighting From Interview Protocols 


Mine eae ne Weighted Weighted 
Sear hsente Weighting | Minimum | Maximum 
Score Score 

Program setup S) 15 x1.60 8 24 
Whole-group instruction 3 9 X2.67 8 24 
Rotation block 8 24 x1.00 8 24 
Whole-group wrap-up 3 2.67 8 24 
Data and reports 3 x2.67 8 24 
Classroom fidelity score 40 120 
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Appendix B 
WDPI/MPS READ 180 Classroom Observation Protocol’? 


Whole-Group Instruction 
(Score Range 7-21) 


Level of Proficiency 


High 


Medium 


Low 


A. Whole-group 
instruction time* 


Whole-group instruction 
is observed for 18-22 
minutes at the beginning 
of the class period. 


Whole-group 
instruction is observed; 
however, it is outside 
the time allotment 
and/or does not take 
place at the beginning 
of the class period. 


Whole-group 
instruction is not 
observed. 


B. Instructional lesson 
on language and 
literacy skills? 


The teacher provides 
explicit instruction or a 
lesson focused on 
developing students’ 
language and literacy 
skills (e.g., vocabulary, 
writing, grammar, usage, 
mechanics). In addition, 
the information is clearly 
communicates it to 
students 


The teacher provides 
explicit instruction or a 
lesson focused on 
developing students’ 
language and literacy 
skills, but does not 
clearly communicate it 
to students 


Teacher does not 
provides explicit 
instruction or a 
lesson focused on 
developing students’ 
language and 
literacy skills during 
this time 


C. Teacher’s use of 


general language 
modeling strategies 


Teacher provides many 
language modeling 
strategies (e.g. models 
fluent reading, models 
reading skills and 
strategies) 


Teacher provides some 
language modeling 
strategies 


Teacher rarely or 
never uses language 
modeling strategies. 


D. Student 
Engagement 


Teacher uses structured 
engagement routines and 
the majority of students 
are actively engaged and 
are participating. 


Teacher uses 
structured engagement 
routines; however, 
some of the students 
participate and are 
actively engaged. 


Teacher does not 
use structured 
engagement 
routines. Few 
students participate 
or are engaged. 


' Observation questions were incorporated information from the Scholastic READ 180 Teacher Implementation 


Guide, CLASS Observation Protocol (Pianta, R. C., La Paro, K. M., & Hamre, B. K., 2008). Classroom Assessment 


Scoring System Manual K-3, Baltimore, MD: Paul H. Brookes), input from the Milwaukee Public Schools READ 
180 Project Coordinator, and previous READ 180 classroom observations by Learning Point Associates prior to its 


merger with AIR. 
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Level of Proficiency 


High Medium Low 
E. Teacher Feedback | Teacher consistently Teacher sometimes Teacher rarely or 

provides explicit feedback | provides explicit never provides 

as opportunities arise. feedback, but misses feedback. 
some opportunities to 
provide feedback. 
Or, feedback teacher 
provides is 


superficial/not specific 
to student behavior or 


performance. 

F. Learning Teacher makes clear Teacher makes Teacher does not 
Connections to connections between connections between make connections 
Rotations whole-group learning whole-group learning | between whole- 

bee . activities and group learning 
activities and rotational : wets See 
oe rotational activities, activities and 
BE but discussion of rotational activities. 
connections are brief 
or unclear. 

G. Learning Teacher makes clear Teacher makes Teacher does not 
Connections to connections between connections between make connections 
Real World whole-group learning whole-group learning | between whole- 


activities and real life activities and real life | group learning 


} experiences, but activities and real 
CAPeLTENn Os discussion of life experiences. 
connections are brief 
or unclear. 


“Tf whole-group instruction is not observed, the classroom should receive low ratings for each of the additional 
items under Whole-Group Instruction. 


> Teacher may also use this time to prepare students for tests, address district standards, discuss goals, and review 
classroom procedures. Therefore, you may score this item High if these activities are observed in place of a lesson. 
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Small-Group Instruction 
(Score Range = 7-21) 


Level of Proficiency 


High 


Medium 


Low 


H. Small-Group 
Instruction Time ‘ 


I. Small Group Size 


J. Planned and 
structured activity 


Gl: 


G3: 
4 Overall: 


All students participate 
in one 18-22 minute 
small-group session. 


All small-group 
sessions include 8 or 
fewer students. 


The teacher 
implements a planned 
and structured activity 
and clearly 
communicates it to 
students. 


Small-group sessions 
are provided; though 
the amount of time for 
all sessions may be 
outside of the specified 
minutes, or some 
students in the 
classroom do not 
participate in a small- 
group time at all. 


Some small-group 
sessions include more 
than 8 students. 


The teacher 
implements a planned 
and structured activity, 
but does not clearly 
communicate it to 
students. 


No students participate 
in a small-group 
session. 


All small-group 
sessions include more 
than 8 students. 


Teacher does not 
implement a planned 
and structured activity 
for this time. 


K. Teacher’s use of 
general language 


Teacher provides many 
language modeling 


Teacher provides some 
language modeling 


Teacher rarely or 
never uses language 


modeling strategies (e.g. models | strategies modeling strategies. 
strategies fluent reading, models 
G1 reading skills and 
G2 strategies) 
G3: 
“ Overall: 
L. Teacher’s use of Teacher frequently Teacher sometimes Teacher rarely or 
differentiated differentiates differentiates never differentiates 


instruction ° 


instruction either 
within or between 
small groups (e.g., 
students working on 
different skills, using 
different modalities, 
teacher directs specific 
questions to specific 
students.) 


instruction either 
within or between 
small groups. 


instruction either 
within or between 
small groups. 
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Level of Proficiency 


High Medium Low 
M. Student Teacher uses Teacher uses Teacher does not use 
Engagement structured engagement | structured engagement | structured engagement 
G1: routines and the routines; however, routines. Few students 
G2 majority of students some of the students participate or are 
G3: are actively engaged participate and are engaged. 
‘——_ | and are participating. i 
a oarall: P paling actively engaged. 
N. Teacher Feedback | Teacher consistently Teacher sometimes Teacher rarely or 
G1 provides explicit provides explicit never provides 
G2 feedback as feedback, but misses feedback. 
C3 opportunities arise. some opportunities to 
r —__ provide feedback. 
Overall: 


(e.g. teacher verbally 
responds to student 
responses; teacher 
provides information 
to class on their 
progress; teacher 
adjusts lesson to meet 
student needs) 


Or, feedback teacher 
provides is 
superficial/not specific 
to student behavior or 
performance. 


“Tf small-group instruction is not observed, the classroom should receive low ratings for each of the additional items 
under Small-Group Instruction. 
“ The overall rating should be determined as follows: High = 3 Highs, or 2 Highs and 1 Medium; Medium = at least 
2 Mediums, or a combination of Highs and Lows; Low = 2 Lows and 1 Medium, or 3 Lows. If only two small-group 
rotations are observed, the rating should be assigned based on the lowest scoring rotation. 
“Following the class period, you may ask the teacher, “How did you plan for your (small group) lessons today?” to 
further inform your rating for this item. 
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Modeled and Independent Reading 
(Score Range =5-15) 


Level of Proficiency 


High 


Medium 


Low 


O. Modeled and 
Independent 
Reading Time ' 


All students participate 
in 18-22 minutes of 
modeled and 
independent reading 
time (includes time for 
writing in logs). 


Time for modeled and 
independent reading is 
provided; though the 
amount of time may be 
outside of the specified 
minutes, or some 
students in the 
classroom do not have 
the opportunity to 
participate in modeled 
and independent 
reading. 


No students are given 
the opportunity to 
participate in modeled 
and independent 
reading. 


P. Student on-task 
behavior using 


The majority of 
students appear to be 


Some of the students 
appear to be listening 


Few or none of the 
students appear to be 


READ 180 listening and following | and following along listening and following 
Audiobooks or along with Audiobooks | with Audiobooks or along with Audiobooks 
Paperbacks or independently independently reading | or independently 
G1: reading for the for the majority of the | reading for the 
G2 majority of the time. time, but other students | majority of the time. 
G3: are off task at times. 
® Overall: 
Q. Student reflections | The majority of Some of the students Few or none of the 
on text students respond to participate respond to__| students respond to 
G1 reading in writing reading in writing reading in writing. 
G2 (e.g., use of logs or 
G3: journals). 
® Overall: 


R. Teacher Awareness 


The teacher 
demonstrates proactive 
behaviors to stay 
aware of students’ 
activities and needs in 
the independent 
reading area the 
majority of the time, 
even when working 
with a small group. 
The teacher anticipates 
problems and plans 
accordingly. 


The teacher 
demonstrates some 
proactive behaviors to 
stay of students’ 
activities and needs the 
independent reading 
area while working 
with a small group, but 
may miss some events 
occurring in the 
independent reading 
area. 


The teacher fails to 
indicate an awareness 
of students’ activities 
and needs in the 
independent reading 
area while working 
with the small group. 
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Level of Proficiency 


High Medium Low 
S. Teacher Teacher responds to Teacher sometimes Teacher does not 
responsiveness to independent reading responds to respond to independent 
questions/problems | group questions or independent reading reading group 
equipment problems. group questions or questions or equipment 
Or, no instances where | equipment problems, problems. 
teacher needed to but fails to respond to 
respond. other issues or 


responds in a manner 
that interrupts the 
activities of the small 
group substantially. 


(e.g. teacher quickly 
addresses any activity 
or equipment problems 
at an appropriate time 
in regards to the small- 
group activities group 
activities) 


If modeled and independent reading is not observed, the classroom should receive low ratings for each of the 
additional items under Modeled and Independent Reading. 


5 The overall rating should be determined as follows: High = 3 Highs, or 2 Highs and 1 Medium; Medium = at least 
2 Mediums, or a combination of Highs and Lows; Low = 2 Lows and 1 Medium, or 3 Lows. If only two Modeled 
and Independent Reading rotations are observed, the rating should be assigned based on the lowest scoring rotation. 
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Computer Rotations 
(Score Range = 5-15) 


Level of Proficiency 


High Medium Low 
T. Instructional All students are given Time for using the No students are given 
Software Time " the opportunity to READ 180 the opportunity to use 


participate in 18-22 
minutes on the 
computer using the 
READ 180 
instructional software. 


instructional software 
is provided; though the 
amount of time may be 
outside of the specified 
minutes, or some 
students in the 
classroom do not have 
the opportunity to use 
the READ 180 
instructional software. 


the READ 180 
instructional software 


U. Student on-task 
behavior using 


The majority of 
students appear to use 


Some of the students 
appear to use the 


Few or none of the 
students appear to use 


READ 180 the READ 180 READ 180 the READ 180 
instructional instructional software | instructional software, | instructional software 
software throughout their although other students | throughout the entire 
Gl rotation. may not use the rotation. 
G2 software throughout 
G3: the entire rotation. 
‘Overall: 
V. Use of multiple The majority of Some of the students Few or none of the 
zones students appear to be appear to be utilizing students are working 
G1 utilizing more than one | more than on zone. on more than one zone. 
Q2 zone at the computers. 
G3: 
"Overall: 


W. Teacher Awareness 


G3: 
‘Overall: 


The teacher 
demonstrates proactive 
behaviors to stay 
aware of students’ 
activities and needs in 
the computer area the 
majority of the time, 
even when working 
with a small group. 
The teacher anticipates 
problems and plans 
accordingly. 


The teacher 
demonstrates some 
proactive behaviors to 
stay of students’ 
activities and needs the 
computer area while 
working with a small 
group, but may miss 
some events occurring 
in the independent 
reading area. 


The teacher fails to 
indicate an awareness 
of students’ activities 
and needs in the 
computer area while 
working with the small 


group. 
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Level of Proficiency 


High Medium Low 
X. Teacher Teacher responds to Teacher sometimes Teacher does not 
responsiveness to computer group responds to computer __| respond to computer 
questions/problems | questions or equipment | group questions or group questions or 
problems. equipment problem, equipment problems 
Gl but fails to respond to 
G2 Or, no instances where other ete Ob 
G3: teacher needed to ae aan Hance ian 
‘"— | respond t at interrupts the 
Overall: activities of the small 


(e.g. teacher quickly 
addresses any activity 
or equipment problems 
at an appropriate time 
in regards to the small- 
group activities) 


group substantially. 


"Tf the computer rotation is not observed, the classroom should receive low ratings for each of the additional items 

under Computer Rotations. 

‘The overall rating should be determined as follows: High = 3 Highs, or 2 Highs and 1 Medium; Medium = at least 
2 Mediums, or a combination of Highs and Lows; Low = 2 Lows and 1 Medium, or 3 Lows. If only two computer 


rotations are observed, the rating should be assigned based on the lowest scoring rotation. 


Whole-Group Wrap-Up 
(Score range = 3-9) 


Level of Proficiency 


High 


Medium 


Low 


Y. Wrap-Up Time 


Whole-group wrap-up 
time is observed for 8- 
12 minutes at the end 
of the class period. 


Whole-group wrap-up 
time is observed; 
however, it is outside 
the time allotment 
and/or does not take 
place at the end of the 
class period. 


Whole-group wrap-up 
time is not observed. 


Z. Teacher Summary 


Teacher reviews key 
points covered in the 
lessons or readings in 
a clear, concise 
manner. 


Teacher reviews key 
points covered in the 
lessons or readings but 
summary is unclear or 
not concise. 


Teacher does not 
review key points 
covered in the lessons 
or readings. 
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Level of Proficiency 


High 


Medium 


Low 


AA, Student 


Reflections 


The majority of 
students reflect on the 
day’s activities, skills 
learned, or material 
read (e.g., students 
summarize their 
activities from the day 
and/or students reflect 
in writing, such as an 
Exit Slip or in a Daily 
Reflection Journal ). 


Some students reflect 
on the days’ activities, 
skills learned, or 
material read. 


Classroom Management and Climate 
(Score Range = 5-15; not part of classroom fidelity score) 


Few or none of the 
students reflect on the 
days’ activities, skills 
learned, or material 
read. 


Level of Proficiency 


High 


Medium 


Low 


BB.Effectiveness in 
addressing 
behavioral issues 


The teacher consistently 
addresses behavioral 
issues. Behavior 
management does not 
take time away from 
learning. No behavioral 
issues are observed. 


The teacher 
inconsistently addresses 
behavioral issues. Few 
times when misbehavior 
continues, escalates, or 
takes time away from 
learning. 


The teacher rarely, it 
ever, addresses 
behavioral issues. 
Frequent instances of 
misbehavior continuing, 
escalating, or taking 
time away from learning. 


CC.Smooth 
transitions 
between all 
models* 


The teacher consistently 
uses a timer, two minute 
warning, and/or other 
method to indicate to 
students of a rotation 
change. 


All transitions are quick 
and efficient. Students 
know what to do and 
there is minimal wasted 
instructional time. 


The teacher sometimes 
uses a timer, two minute 
warning, and/or other 
method to indicate to 
students of a rotation 
change. 


Some instructional time 
may be wasted and some 
students seem confused 
about how to transition 
into a new activity. 


The teacher never uses a 
timer, two minute 
warning, and/or other 
method to indicate to 
students of a rotation 
change. 


Few or none of the 
transitions are quick and 
efficient. Instructional 
time is lost during most 
or all transitions. 
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Level of Proficiency 


High 


Medium 


Low 


DD. Clear 
instructions and 
outline of 


expectations 


Instructions are 
consistently outlined and 
clarified. 


The majority of students 
appear to understand 
expectations the majority 
of the time. 


Instructions are 
sometimes outlined and 
clarified. 


The majority of students 
appear to understand 
expectations some of the 
time; or, some students 
appear to understand 
expectations the majority 
of the time. 


Instructions are rarely 
or never outlined and 
clarified. 

Student behavioral 
expectations are not 
posted. 


The majority of students 
look confused as to 
expectations for the 
majority of the time. 


EE. Mutual Respect 
among teacher 
and students 


Teacher and students 
demonstrate respect for 
one another the majority 
of the time.(e.g., using 
respectful language, eye 
contact, cooperation) 


Teacher and students 
demonstrate respect for 
one another some of the 
time. 


Teacher and students 
rarely or never 
demonstrate respect for 
one another. 


FF. Disrespectful 
behaviors among 
teacher and 
students 


No instances, or very 
mild instances, of 
disrespect are observed. 


Some instances of 
disrespect between 
teacher and/or students. 


Frequent instances of 
disrespect between 
teacher and/or students 
are observed. (e.g. 
sarcastic remarks, 
teasing, humiliation) 


“Tf the teacher does not implement READ 180 rotations, this item should rated based on other transitional times 
(e.g., beginning of class period). 


Materials (Score Range = 7-21; not part of classroom fidelity score) 


GG. 


Quantity of 
READ 180 Paperbacks” 


Level of Proficiency 


High 
There are a sufficient 
number of READ 180 
paperback books at 
various levels 
available and 


accessible to students. 


Medium 


READ 180 paperback 
books at various levels 
are available and 
accessible to students; 
the number of books at 
one or more levels is 
insufficient. 


Low 


READ 180 paperback 
books at various levels 
are not available and 

accessible to students. 
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Level of Proficiency 


useable condition and 
are organized in a way 
that is appealing to 
students (e.g. arranged 
by reading level and 
on shelf with spine 
facing out). 


are in useable 
condition, but are 
poorly organized and 
not accessible or 
appealing to students. 


High Medium Low 
HH. = Quality/ The majority of READ | The majority of the The majority of the 
Organization of Books | 180 paperback and READ 180 paperback | READ 180 paperback 
Audiobooks are in books and Audiobooks | books and Audiobooks 


are in poor or 
unusable condition 
and are not organized 
to encourage student 
use. 


II. Quantity of 
READ 180 
Audiobooks? 


There are a sufficient 
number of Audiobooks 
available and 
accessible to the 
majority of students. 


Some Audiobooks are 
available and 
accessible to students; 
however, the number 
is insufficient. 


Few/no students have 
access to Audiobooks 
because they are either 
damaged or the 
number is insufficient. 


JJ. Quantity of 
rBooks” 


The majority of 
students have an 
rBook. 


Some students have an 
rBook, while other 
students do not 
(students w/o rBook 
have a notebook). 


Few students or none 
have an rBook or 
notebook. 


KK. Functional 
Computers and READ 
180 Software 


The majority of 
students have access to 
a functional computer 
with headphones, 
microphone, and 
working software. 


Some students have 
access to a functional 
computer with 
headphones, 
microphone, and 
working software; 
other students do not. 


Few or none of the 
students have access to 
a functional computer 
with headphones, 
microphone, and 
working software. 


seating in the reading 
area. 


the reading area. 


LL. Functioning All students that would | Some students have Few or none of the 
CD players” like access to a CD access toa CD player; | students have access to 
player, have access. at least one student a CD player that 
does not have access would like access. 
that would like access. 
MM. Comfortable There is a sufficient There is some There is no 
Seating amount of comfortable | comfortable seating in | comfortable seating in 


the reading area. 


» Students may need to ask teachers for CD player or other listening device. 
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Classroom Fidelity Score Weighting From Observation Protocol 


Min. | Max. ee ils eats ego 
Weighting | Minimum | Maximum 
Score | Score 
Score Score 
Whole-Group Instruction 7 21 x1.00 7 21 
Small-Group Instruction 7 21 x1.00 7 21 
Modeled and Independent Reading 5 15 x1.40 7 21 
Computer Rotation fs) 15 x1.40 7 21 
Whole-Group Wrap-up 3 9 x2.33 7 21 
Classroom Fidelity Score 35 105 
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Quantity 


Appendix C 
WDPI/MPS Program Implementation Fidelity Key 
(Professional Development and Instructional Supports) 


Saar: Quantity Ratings Quality Source Quality Ratings 
High (3) Medium (2) Low (1) High (3) Medium (2) Low (1) 
READ 180 PD Logs Attended 2 Attended 1 Attended 0 | Interviews: Teachers took a lot Teachers took Teachers took 
Training days day days Fall Tchr: 33, of information away | some information | little 
(Or, 33a, 41, 42 from the trainings, away from the information 
experienced Principal: 25, and/or reported that | trainings and/or away from the 
READ 180 25a, 26, 26a it was of good or reported that it trainings and/or 
teacher) excellent quality. was of fair reported that it 
(Or, experienced quality. was of poor 
READ 180 teacher) quality. 
READ 180 PD Logs Attended Attended Attended Interviews: Teachers took a lot Teachers took Teachers took 
Roundtables 75% or 50-74% of less than Fall Tchr: 36, of information away | some information | little 
more sessions 50% of 36a, 41, 42 from the teacher away from the information 
sessions sessions Spring Tchr: 42, | roundtables, and/or teacher away from the 
42a, 47, 48 reported that they roundtables, and teacher 
Principal: 25, were of good or /or reported that roundtables, 
25a, 26, 26a, 27 | excellent quality. they were of fair and/or reported 
quality. that they were 
of poor quality. 
READ 180 PD Logs Completed Completed Completed | Interviews: Teachers took a lot Teachers took Teachers took 
RED Course 75% or 50-74% of less than Fall Tchr: 34, of information away | some information | little 
online more of required 50% of 34a, 41, 42 from the online away from the information 
required lessons required Spring Tchr: 40, | courses, and/or courses, and /or away from the 
lessons lessons 40a, 47, 48 reported that it was reported that it courses, and/or 
Principal: 25, of good or excellent | was of fair reported that it 
25a, 26, 26a quality. quality. was of poor 
quality. 
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Quantity 


excellent quality. 
(Or, already 
licensed) 


were of fair 
quality. 


Sance Quantity Ratings Quality Source Quality Ratings 
High (3) Medium (2) Low (1) High (3) Medium (2) Low (1) 
Mentoring Coach Received 7- | Received 6- | Received Interviews: The Scholastic/Read | The The 
Support from Logs 9 coaching 4 coaching 0-3 Fall Tchr: 30, 32 | 180 representative Scholastic/Read Scholastic/Read 
Scholastic/ sessions sessions coaching Spring Tchr: 36, | has affected Read 180 representative | 180 
READ 180 sessions 38, 39 180 instruction toa | has affected Read | representative 
representative Principal: 21, 23, | large extent. 180 instruction to | has affected 
24 a moderate extent. | Read 180 
instruction to a 
small extent or 
not at all. 
SAM Data PD logs Attended ——— Did not Interviews: Teachers fully Teachers Teachers did 
training (Or, attend Fall Tchr: 37, participated in participated in not participate 
experienced 37a, 41, 42 offered SAM data offered SAM data | in offered SAM 
READ 180 Spring Tchr: 43, | training, took a lot training, took data training, 
teacher) 43a, 47, 48 of information away | some information | took little 
Principal: 25, from the training, away from the information 
25a, 26, 26a and/or reported that | training, and /or away from the 
it was of good or reported that it training, and/or 
excellent quality. was of fair reported that it 
quality. was of poor 
quality. 
Graduate Project Completed Enrolled but | Not Interviews: Teachers Teachers Teachers did 
courses Coordinato | (Or, already | not enrolled Fall Tchr: 38, participated in participated in not participate 
r Records licensed) completed and needed | 38a, 41, 42 graduate courses graduate courses in any graduate 
Spring Tchr: 44, | related to Read 180 | related to Read courses related 
44a, 47, 48 instruction, took a 180 instruction, to Read 180 
Principal: 25, lot of information took some instruction, took 
25a, 26, 26a away from the information away | little 
graduate courses, from the graduate | information 
and/or reported that | courses, and /or away from the 
they were of good or | reported that they | graduate 


courses, and/or 
reported that 
they were of 
poor quality. 
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Quantity 


Sance Quantity Ratings Quality Source Quality Ratings 
High (3) Medium (2) Low (1) High (3) Medium (2) Low (1) 
READ 180 PD logs Attended Did not Interviews: Teachers fully Teachers Teachers did 
conference attend Fall Tchr: 35, participated in participated in not participate 
(Year 2 only) 35a, 41, 42 offered Read 180 offered Read 180 | in offered Read 
Spring Tchr: 41, | seminars or seminars or 180 seminars or 
41a, 47, 48 conferences, took a conferences, took | conferences, 
Principal: 25, lot of information some information | took little 
25a, 26, 26a away from the away from the information 
seminars/conference | seminars/conferen | away from the 
s, and/or reported ces, and /or seminars/confer 
that it was of good reported that it ences, and/or 
or excellent quality. | was of fair reported that it 
quality. was of poor 
quality. 
Materials Interviews: | All materials | All materials | Materials Uf, Vf “Wf, Uf 
Available / Fall Tchr: are available | are are 
Need 10, 15 and were available, missing, 
additional Spring available on | but there there was a 
materials Tchr: 13, time. There | wasadelay | delay in 
14 is a clear in materials. | materials, 
Principal: process to There is a and/or 
9, 10, 13 acquire somewhat there is not 
more clear process | a clear 
materials. to acquire process to 
more acquire 
materials. more 
materials. “iy, Yp “Wf “Wf 
Quantity of Interviews: | The majority | Some Few A “Hy Vf “Yj 
rBooks Fall Tchr: of students students students or 
11, 16 have their have their none have 
Spring own rBook own rBook, | their own 
Tchr: 9, 14, while other rBook 
15 students do 
Principal: not 
10 
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Pree Quantity Ratings Quality Source Quality Ratings 
High (3) Medium (2) Low (1) High (3) Medium (2 Low (1 
Quantity of Interviews: | Therearea | READ 180 | READ 180 WH 
READ 180 Fall Tchr: sufficient paperback paperback 
Paperbacks 12, 12a, 16 | number of books at books at 
Spring READ 180 various various 
Tchr: 10, paperback levels are levels are 
10a, 14,15 | books at available not 
Principal: various and available 
10 levels accessible to | and 
available students; accessible 
and however, the | to students. 
accessible to | number of 
students. books at one 
or more 
levels is 
insufficient “yy, 
Quantity of Interviews: | There area | Audiobooks | There are Vf 
READ 180 Fall Tchr: sufficient are available | no 
Audiobooks 12, 16 number of and Audiobook 
Spring Audiobooks | accessible; s available 
Tchr: 10, available however, the | and 
14,15 and number is accessible 
Principal: accessible to | insufficient | to students. 
10 students. ) Yi 
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Appendix D 
Impact Analysis Methods 


The statistical analysis of student outcomes focuses on the impact of READ 180 on students’ 
reading achievement and engagement. The intervention targeted students in Grades 6—10 and 
focused primarily on the impact of one year of READ 180 for the treatment group students. The 
initial study (prior to the cancelation of the Striving Readers grant) was to assess the overall 
impact of a two-year READ 180 intervention, with statistical power gained through the pooling 
of effects across grades and cohorts (i.e., the samples will be combined by standardizing the 
outcome score for each grade). In addition to the main impact estimate (across all grades), we 
also fit models to study the impact of READ 180 by both school level—middle school (Grades 
6-8) versus high school (Grades 9—10)—and by student subgroup (ELL and special education 
students). 


Model Specifications: Cross-Sectional Model 


To measure the impact of the READ 180 intervention on reading proficiency for students in the 
five MPS study schools, we tested for both intent-to-treat (ITT) and treatment-on-the-treated 
(TOT) effects. Although both models are considered in our analysis, the primary model used for 
reporting the effectiveness of the READ 180 intervention is the ITT analysis. The ITT group 
includes all students randomized to the study who enrolled in the school and grade level at which 
they were randomized in the 2010-11 school year. We examined and presented three ITT 
models, defined as follows: 


(1) Yi = Bo + BiTy + ZBISGRg + ej 
(2) Yy = Bo + BiTy + ZPYSGRy + LPP; + ej 


In these models, Y;; represents the student-level, postintervention outcome (e.g., reading 
achievement on the MAP reading assessment completed in June 2011). Tj is an indicator of the 
treatment status of student i in school j, P; represents two pretest standardized test scores (i.e., 
WCKE and ThinkLink), D; represents student-level covariates (e.g., gender, ethnicity, economic 
status, special education status, and ELL status), and SGR,; represents a school-by-grade-by- 
randomization block for a given student (to account for the stratified random sampling 
procedures). In addition, models include missing data indicators for prior achievement scores 
(equaling 1 for students with missing WKCE or ThinkLink scores, respectively). 


The TOT analysis was conducted to account for the possibility of treatment crossover and other 
forms of treatment noncompliance. The TOT estimate accounts for that noncompliance through 
the use of a two-stage least-squares approach, using random assignment as an instrumental 
variable for treatment receipt (Angrist, Imbens, & Rubin, 1996). The second-stage model for the 
TOT analysis is 


(4) Yj j- Bo a BiR ii oF XBijSGRj + XBiPi t XPD; t &ij 
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In this model B; is the estimated impact of READ 180 for a student i who enrolled in READ 180 
(R’) based on their assignment to the treatment group. 


Selection of Covariates 


Several student-level covariates were be available for inclusion in the impact models. The power 
analysis was conducted using an assumption that included covariates would explain 50 percent 
of the variance in student scores. This variance reduction assumption was based primarily on the 
inclusion of student pretest scores in the impact model. Several additional student covariates, 
however—measured prior to randomization—were considered for model inclusion. These 
covariates were gender, ethnicity, economic status, ELL status, and special education status. 


The impact models were fit from three perspectives. First, a basic model including treatment 
status as the only independent variable (the pure experiment) was considered. Second, pretest 
was added to the model to increase precision and adjust for any baseline differences in student 
ability. Third, a full model with all possible covariates was fit. The decision criteria for including 
covariates for the third model utilized a backward selection criterion with a critical value of 0.2 
(t-statistic significance) for inclusion. 


Although a series of statistical models was fit to the data, the evaluators understand the U.S. 
Department of Education is interested in a single treatment effect estimate. Therefore, since the 
results do not vary dramatically among the three modeling approaches, the estimate of the ITT 
treatment effect from the covariates-included model (the third modeling approach) will be used 
as the estimate of the effect of two years of READ 180 treatment. 


Treatment of Missing Data 


There were two types of missing data that occurred during our analysis of program impact: 
missing outcome data and missing covariate data. Missing outcome data was not imputed and 
therefore all models fit only those cases where students had a posttest score for the NWEA MAP 
assessment. Missing covariates were treated differently. In particular, some students did not have 
either a WKCE reading score or a ThinkLink benchmark score (the two assessments used to 
determine eligibility for random assignment). To account for this missingness in the data, we 
created indicator variables that were coded as 1 if the student was missing the prescore and 0 
otherwise. In addition, the missing value for the prescore was coded to 0. This procedure keeps 
all cases in the analysis (i.e., no exclusions due to missing data), and does not lead to biased 
results because students were randomized into the treatment conditions. All three impact models 
include the same number of students in the analysis. 
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Appendix E 
Student Engagement Survey 


Student Responses to Self Efficacy Survey Items 


oe, : Somewhat | Somewhat 


16.9% 53.4% 25.4% 
21.7% 55.0% 16.3% 


ane eee 
complete my school work. N Disagree Disagree Agree Agree 


15. My reading ability is above Somewhat | Somewhat 
average. N Disagree Disagree Agree Agree 


14.5% 27.4% 37.6% 20.5% 


Treatment 11.7% 23.4% 43.0% 21.9% 


; Somewhat | Somewhat 


Control 10.3% 36.8% 49.6% 
Treatment 33.9% 54.3% 


Fa DI bho BE 
graduate from high school. N Disagree Disagree Agree Agree 


RL aih wood weade: Somewhat | Somewhat 
: 8 : N Disagree Disagree Agree Agree 


Control 43.1% 38.8% 
Treatment 34.6% 56.7% 


Student Responses to Behavioral Engagement Survey Items 


a Somewhat | Somewhat 


Control 21.2% 18.6% 37.3% 22.9% 
Treatment 14.7% 20.9% 34.1% 30.2% 


[arses a BOO hoi hewn 
articles to read on my own time. N Disagree Disagree Agree Agree 
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a ee 
assignments for school. N_ | Disagree Disagree Agree Agree 


116 | 27.6% 26.7% 28.4% 17.2% 
129 | 24.8% 20.2% 34.9% 20.2% 


Leal beg 
Disagree Agree Agree 
27 


25. If I am reading an interesting book 
or article I sometimes lose track of Somewhat | Somewhat 
time. N Disagree Disagree Agree Agree 


115 | 13.0% 19.1% 33.9% 33.9% 
127 | 10.2% 14.2% 30.7% 44.9% 


Student Responses to Emotional Engagement Survey Items 


3. I feel comfortable reading aloud in Somewhat | Somewhat 

front of other classmates. Agree 
117 24.8% 
130 32.3% 


6. If my teacher told me I was a good Somewhat | Somewhat 

reader, I would feel proud. Agree 
68.1% 
130 70.0% 


8. IL enjoy reading in my free time. ? omen nat Somewoat 
N Disagree Disagree Agree Agree 


117 | _ 25.6% 31.6% 28.2% 14.5% 
130 | 26.9% 23.8% 34.6% 14.6% 


12. If my parents told me I was a good Miece lipoma Peemneees lenny 

reader, I would feel proud. N Disagree Disagree Agree Agree 
64.7% 
66.4% 


13. Reading information online is fun. ; Spmewnat Someyiat 
N Disagree Disagree Agree Agree 


117 | __ 17.1% 23.1% 37.6% 22.2% 
129 | 16.3% 20.2% 35.7% 27.9% 
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ee ee 
in what I am reading. N Disagree Disagree Agree Agree 


i a ea 
example: novels and fantasy). N Disagree Disagree Agree Agree 


example: biographies and history). N Disagree Disagree Agree Agree 


Faleare ge  N nd hed 
can sometimes be interesting. N Disagree Disagree Agree Agree 


Student Responses to Cognitive Engagement Survey Items 


4. I sometimes summarize what I’ve 

read to help me better understand 

important information in the reading Somewhat | Somewhat 

material. Disagree Disagree Agree Agree 
16 


117 23.9% 33.3% 26.5% 
129 | 17.8% 15.5% 31.8% 34.9% 


7. If what I am reading becomes 

difficult, sometimes I try to pay closer 

attention so I can better understand Somewhat | Somewhat 

what it means. Disagree Agree Agree 


36.4% 50.0% 


30 29.2% 56.2% 


116 33.6% 59.5% 
1.6% 31.3% 62.5% 


14. Being a good reader will be 
important after I graduate from high Somewhat | Somewhat 
school. N Disagree Disagree Agree Agree 


116 22.4% 71.6% 
128 18.8% 72.7% 


American Institutes for Research Striving Readers Impact Study—101 


19.5% 14.2% 38.1% 28.3% 
19.8% 12.7% 40.5% 27.0% 


11.3% 49.6% 34.8% 
12.7% 40.5% 41.3% 


11.1% 14.5% 32.5% 41.9% 
11.1% 33.3% 48.4% 
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